Paper deep dive
ReproAgent: Contract-Guided Paper-to-Code Reproduction
Xue Hu, Zewei Pan, Zhongyuan Wang, Zhou Liu, Zeli Su, Wentao Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/29/2026, 4:19:10 AM
Summary
The paper introduces ReproAgent, a four-stage pipeline (Prepare-Plan-Generate-Repair) for paper-to-code reproduction. It utilizes a persistent implementation contract with two channels: an implementation-requirement channel to extract code obligations from paper snippets, and a reference-evidence channel to retrieve content and structure evidence from related repositories. Evaluated on PaperBench Code-Dev, ReproAgent achieves the highest mean scores among same-backbone scaffolds using Claude-Sonnet-4.5 and Gemini-3-Flash.
Entities (9)
Relation Signals (8)
ReproAgent → evaluatedon → PaperBench Code-Dev
confidence 95% · On PaperBench Code-Dev, ReproAgent reaches the highest mean score
ReproAgent → uses → Reference-evidence channel
confidence 95% · ReproAgent... built around a persistent implementation contract with two channels: ... a reference-evidence channel
ReproAgent → uses → Implementation-requirement channel
confidence 95% · ReproAgent... built around a persistent implementation contract with two channels: an implementation-requirement channel
ReproAgent → backedby → Gemini 3 Flash
confidence 90% · under both Claude-Sonnet-4.5 and Gemini-3-Flash
ReproAgent → backedby → claude-sonnet-4.5
confidence 90% · under both Claude-Sonnet-4.5 and Gemini-3-Flash
PaperBench Code-Dev → partof → PaperBench
confidence 90% · PaperBench (Starace et al., 2025)... PaperBench Code-Dev (Starace et al., 2025)
ReproAgent → outperforms → BasicAgent
confidence 85% · ReproAgent reaches the highest mean score among same-backbone scaffolds... BasicAgent
ReproAgent → outperforms → DeepCode
confidence 85% · ReproAgent reaches the highest mean score among same-backbone scaffolds... DeepCode
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Paper-to-code reproduction asks scientific AI agents to turn research papers into executable repositories that preserve the paper's method, protocol and artifacts. This is difficult because the specification is split: explicit paper content such as algorithms, metrics and artifacts is often lost across long agent trajectories, while implicit details such as framework defaults and conventions inherited from related work are absent from the paper. We introduce ReproAgent, a four-stage Prepare--Plan--Generate--Repair pipeline built around a persistent implementation contract with two channels: an implementation-requirement channel that turns paper snippets into code obligations, and a reference-evidence channel that retrieves content and structure evidence from related repositories. Both are bound to work packages, projected into file-level contracts, and consumed across generation and repair. On PaperBench Code-Dev, ReproAgent reaches the highest mean score among same-backbone scaffolds under both Claude-Sonnet-4.5 and Gemini-3-Flash. End-to-end channel ablations and per-paper cases support the contribution of both channels. Code and experimental artifacts are publicly available.
Tags
Links
- Source: https://arxiv.org/abs/2608.24291v1
- Canonical: https://arxiv.org/abs/2608.24291v1
Trouble viewing inline? Open PDF directly →
Full Text
83,373 characters extracted from source content.
Expand or collapse full text
ReproAgent: Contract-Guided Paper-to-Code Reproduction Xue Hu* Affiliation: Beihang University Email: kernel@buaa.edu.cn Zewei Pan* Affiliation: Shanghai Jiao Tong University Email: pzp0057@sjtu.edu.cn Zhongyuan Wang Affiliation: Beihang University Email: zywang111@buaa.edu.cn Zhou Liu Affiliation: Peking University Zeli Su Affiliation: Minzu University of China Wentao Zhang† Email: zhouliu25@stu.pku.edu.cn, rickamorty@muc.edu.cn, wentao.zhang@pku.edu.cn*Equal contribution. †Corresponding author. Affiliation: Peking University Affiliation: Zhongguancun Academy Affiliation: Beijing Key Laboratory of Data Intelligence and Security (Peking University) Abstract Paper-to-code reproduction asks scientific AI agents to turn research papers into executable repositories that preserve the paper’s method, protocol and artifacts. This is difficult because the specification is split: explicit paper content such as algorithms, metrics and artifacts is often lost across long agent trajectories, while implicit details such as framework defaults and conventions inherited from related work are absent from the paper. We introduce ReproAgent, a four-stage Prepare–Plan–Generate–Repair pipeline built around a persistent implementation contract with two channels: an implementation-requirement channel that turns paper snippets into code obligations, and a reference-evidence channel that retrieves content and structure evidence from related repositories. Both are bound to work packages, projected into file-level contracts, and consumed across generation and repair. On PaperBench Code-Dev (Starace et al., 2025), ReproAgent reaches the highest mean score among same-backbone scaffolds under both Claude-Sonnet-4.5 and Gemini-3-Flash. End-to-end channel ablations and per-paper cases support the contribution of both channels. Code and experimental artifacts are publicly available.11 1 https://github.com/kernel-14/ReproAgent. 1 Introduction Figure 1: Overview of ReproAgent. A four-stage pipeline, Prepare, Plan, Generate, and Repair, carries an implementation-requirement channel and a reference-evidence channel into a shared file-level implementation contract. The implementation-requirement channel preserves paper-derived code obligations across generation and repair, while the reference-evidence channel supplies package-local content and structure evidence from related repositories. Large language models are increasingly used to assist scientific research, but a long-standing reproducibility crisis persists (Baker, 2016; Pineau et al., 2021). Systematic reviews of NLP reproduction studies report that only a small fraction of attempted reproductions recover the original scores (Belz et al., 2021), and even strong LLM judges detect only 46.7% of real-world discrepancies between papers and their accompanying code (Baumgärtner and Gurevych, 2026). Together, faster method production and persistent reproducibility gaps make it harder to validate new methods, compare baselines fairly, and build on prior work. An automated framework that turns each new paper into a faithful, runnable repository, rather than merely executable code, would make paper-specific methods, protocols, and artifacts easier to inspect and reuse. Recent paper-to-code agents take a step in this direction (Seo et al., 2026; Xiang et al., 2025; Zhao et al., 2025; Zhou et al., 2025), but a substantial fidelity gap remains. A recurring failure mode is runnable but unfaithful code: a generated repository may import correctly, pass smoke tests, and even produce plausible experimental results, while still dropping or misimplementing key paper-specific details, such as substituting a generic training loop for the paper’s algorithm, omitting the decisive metric or artifact, or altering a loss term. Runtime success alone is therefore an insufficient proxy for faithful paper reproduction. We identify a split-specification problem behind these failures. A paper states some implementation requirements explicitly, including algorithms, metrics, artifacts and evaluation protocols. However, many decisions needed to build a faithful repository are implicit in framework defaults, cited implementations and community conventions. Existing paper-to-code agents mainly condition generation on the paper text and transient planning context, without a persistent object that carries paper-derived obligations or binds implicit details to external evidence. As a result, explicit requirements drift out of context across long trajectories, while implicit details fall back on model priors rather than grounded evidence. Faithful reproduction therefore needs a persistent implementation contract that preserves paper-derived obligations and attaches external repository evidence to the implementation decisions it supports. ReproAgent instantiates this idea with a global implementation contract containing two channels. The implementation-requirement channel extracts explicit paper knowledge as code obligations, while the reference-evidence channel grounds implicit repository knowledge in related implementations. Both channels are bound to work packages and projected into file-level contracts, so the same contract guides generation and serves as the audit target for detecting missing obligations, unsupported implementation choices, broken interfaces, and runtime failures. ReproAgent carries this contract through a four-stage pipeline. Prepare segments the paper, extracts implementation requirements, and builds a reference-repository index. Plan organizes requirements into work packages, binds package-local reference evidence, derives the target architecture, and freezes file-level contracts over obligations, evidence, validation checks, and file responsibilities. Generate produces the repository file by file under these local contracts. Repair audits the result against the same contract together with runtime checks, so failures are reported as violated paper-derived obligations, missing evidence use, broken interfaces, or execution failures rather than only as crashing commands. We evaluate ReproAgent on PaperBench Code-Dev (Starace et al., 2025), using repository-level rubric grading to test whether generated code implements the target paper rather than merely executing. We run ReproAgent with two backbones, Claude-Sonnet-4.5 and Gemini-3-Flash, and under each backbone ReproAgent reaches the highest mean score among same-backbone scaffolds. Ablations of the two contract channels together with a per-paper case analysis support that both the implementation-requirement and reference-evidence channels contribute to faithful reproduction. Our main contributions are: • We formulate paper-to-code reproduction as contract-guided repository construction under partial paper specifications, where faithful generation requires preserving explicit paper obligations and grounding implicit implementation choices. • We introduce a two-channel implementation contract over paper-derived requirements and related-repository evidence, with work-package bindings, file-level obligations, provenance records, and typed requirement-review/runtime reports that preserve both traces across generation and repair. • We instantiate this contract in ReproAgent, a four-stage Prepare–Plan–Generate–Repair pipeline, and evaluate it on the full 20-paper PaperBench Code-Dev suite with same-backbone comparisons and ablations that remove each contract channel. 2 Related Work LLM agents for research and code. LLM agents have been applied to research workflows (Baek et al., 2024; Lu et al., 2024; Schmidgall et al., 2025), ML engineering (Chan et al., 2024; Tang et al., 2024), and code-generation or repository-level coding settings (Qian et al., 2024; Islam et al., 2024; Hong et al., 2024; Yang et al., 2024; Xia et al., 2025; Zhang et al., 2024). Paper reproduction differs because the target is a scientific procedure described only partially in natural language: the repository must be faithful to paper-specific methods, protocols, and artifacts, not merely pass a generic execution check. Paper-to-code reproduction. Following DLPaper2Code (Sethi et al., 2018), recent systems extend reproduction to whole repositories: PaperCoder, the system introduced by Seo et al. (Seo et al., 2026), stages planning, analysis, and coding; Sci-Reproducer (Xiang et al., 2025) highlights missing-algorithm bottlenecks; AutoP2C (Lin et al., 2025) and HiRAS (Hong et al., 2026) push multimodal parsing and hierarchical supervision; AutoReproduce (Zhao et al., 2025) retrieves along citation graphs; DeepCode (Li et al., 2025b) treats reproduction as agentic coding. To our knowledge, prior systems do not make both paper-derived obligations and retrieved repository evidence a persistent, file-level audit object shared across planning, generation, and repair, which is the traceability layer ReproAgent targets. Reference retrieval and contracts. Retrieval-augmented code generation depends on retrieval granularity (Chen et al., 2024) and pulls evidence from documentation (Zhou et al., 2023; Pimparkhede et al., 2024), repositories (Zhang et al., 2023; Cheng et al., 2024), or code graphs (Li et al., 2025a), but does not tie each snippet to the scientific requirement it supports. Our contract view connects to requirements traceability and design by contract (Gotel and Finkelstein, 1994; Meyer, 1992; Cleland-Huang et al., 2014), adapted to settings where requirements are extracted from paper snippets rather than written formally. ReproAgent therefore splits reference evidence into content evidence for reuse or adaptation and structure evidence for architecture planning, and binds both to work-package and file-level contracts that persist across the pipeline. Evaluation and fidelity. PaperBench (Starace et al., 2025) grades reproductions against rubrics, with related benchmarks studying reproducibility and paper-code mismatch (Yan et al., 2025; Siegel et al., 2024; Baumgärtner and Gurevych, 2026). RePro (Zhou et al., 2025) applies paper-derived criteria as post-hoc refinement, after a candidate repository exists. ReproAgent differs by placing the same paper-derived requirements into the forward generation contract, so the audit target is fixed before generation rather than reconstructed afterwards. Figure 2: ReproAgent overview. The implementation-requirement channel extracts code-facing obligations from the target paper, while the reference-evidence channel retrieves content and structure evidence from related repositories. The shared contract binds both channels into work packages, global and file-level contracts, generation, and validation, yielding a repository that is checked for both requirement fidelity and runtime correctness. Representative unit and evidence records appear in Appendix A. 3 Method ReproAgent turns a paper into a repository by maintaining two persistent contract channels, shown in Figure 2. The implementation-requirement channel preserves what the target paper explicitly says must be implemented. The reference-evidence channel supplies evidence for implicit repository knowledge: implementation conventions, file structure, entry points, and protocol details from cited or related repositories. Both channels are bound to work packages and projected into file-level contracts, so information extracted early remains visible during planning, generation, validation, and repair. A work package is a small planning unit that bundles related requirements with the evidence supporting them; we give its full schema in §3.3. 3.1 Implementation Requirements An implementation requirement is a paper snippet that must be realized in code, and we represent it as a unit u=(id,src,stmt,cite,surf,art,status).u=(id,src,stmt,cite,surf,art,status). Here idid is a stable identifier, srcsrc records the paper source span the unit is drawn from, and stmtstmt is a normalized statement of the obligation. The remaining four fields specify how the unit travels through the rest of the pipeline: citecite stores explicit reference cues when they are present in the paper, and the reference-evidence channel attaches prior-work snippets through unit IDs, source anchors, and match records (§3.2); surfsurf names the expected code surfaces, such as a model class, loss function, optimizer, metric, CLI option, config field, training script, or artifact writer; artart enumerates the artifacts the unit should produce; and statusstatus is a lifecycle flag that stays active until the unit is realized. A concrete example of such a unit is given in Appendix A.1. Extraction. We first segment the paper into section-level chunks, falling back to paragraph-level splits when a section is too long for a single LLM pass. From each chunk, we extract only snippets that imply concrete implementation work, such as algorithmic steps, loss formulations, optimizer and training settings, dataset and preprocessing rules, and reported metrics or artifacts; pure motivation, related work narration, and discussion are skipped. Preservation invariant. To make this preservation operational, we attach a coverage invariant to every active unit so that any silent loss surfaces as a checkable failure inside the pipeline. For every active unit u∈activeu _active, the contract requires own(u),proj(u),prov(u)≠∅, (u),\,proj(u),\,prov(u)≠ , verdict(u)∈, (u) , where Y is the fixed verdict set returned by requirement review, with four possible values: pass, missing, wrong-surface and inconsistent-with-source. The four predicates correspond to four pipeline stages: • Ownership. Plan assigns the unit to at least one work package. • File projection. File planning assigns the unit to at least one file contract whose declared surfaces cover the unit’s expected surfaces. • Provenance. Generation emits a record linking the produced file region back to the unit. • Verdict. Requirement review returns a value in Y. A failed predicate creates a contract violation report and a repair obligation; if the repair budget is exhausted, the unresolved verdict is retained in the diagnostics rather than silently absorbed. 3.2 Reference Evidence Papers often leave implementation details implicit: a paper may cite a prior method without restating its update rule, or refer to a benchmark’s “standard” data augmentation whose canonical form lives in a reference repository. ReproAgent therefore collects reference evidence from related repositories listed in the paper’s bibliography, and represents each evidence item as e=(repo,path,region,units,role,basis).e=(repo,path,region,units,role,basis). Here reporepo identifies the source repository, pathpath the file inside it, and regionregion the matched line range; unitsunits lists the requirement IDs the snippet supports, rolerole records whether the snippet is meant for local reuse or for adaptation as an interface pattern, and basisbasis is a structured match record explaining why the snippet is relevant. Each item is bound to the work package that owns its supported units, so reference code enters generation through a specific package contract rather than as global context. A concrete evidence item is given in Appendix A.2. Content evidence. For each candidate reference repository, ReproAgent indexes file paths, imports, exported symbols, scripts, configurations, and entry points described in the README. A snippet is retained when one of three signals connects it to an owned unit, an expected artifact, or a declared implementation surface: a symbol or interface name match, a path or configuration cue, or a structured LLM judgment. The judgment is stored in basisbasis as the supported unit IDs, the matching surface or artifact, and whether the snippet should be reused locally or adapted as an interface pattern, rather than as a free-form note. The rolerole field then acts as a use constraint: local reuse covers reusable formulas or transforms, while interface adaptation covers structural patterns that should be re-implemented under the package’s own interface, not copied wholesale. Structure evidence. Reference repositories also expose where code should live: file trees, entry scripts, module boundaries, configuration conventions, dependency files, and technology-stack hints. Architecture planning combines these structural priors with the task and requirements to decide the target file tree, stable interfaces, entry points, and dependency graph. Content evidence therefore informs how to implement an obligation, while structure evidence informs where the implementation should live. 3.3 Work Packages and the Global Contract A work package is the local planning unit that binds requirements to evidence. Each package records its goal, the requirement IDs it owns, its dependencies on other packages, the public interface it exposes, the artifacts it should produce, and the evidence items attached to it. After evidence binding, the planner freezes a global contract =(,,ℰ,,),G=(U,\,W,\,E,\,A,\,V), where U is the active requirement set, W the work packages, ℰE the package-bound evidence, A the target architecture, and V the validation plan. Once G is frozen, later stages can refine local implementation choices and add derived helper files, but the active requirements, package ownerships, and evidence-package bindings carried by G stay fixed. File planning then projects G into file contracts F=cfC_F=\c_f\, where each cfc_f specifies its target path, owner package, owned units, imported and exported symbols, attached evidence, produced artifacts, and review checks. Every generated file therefore has a per-file specification grounded in the global contract. 3.4 Generation, Validation, and Repair Generation proceeds file by file in dependency order. Each call is conditioned on the file contract cfc_f, the evidence attached to its owner package, and the already-generated dependency files needed for the local interface; reference code from sibling packages is not injected as global context. This package-local scoping keeps each produced file auditable: a training loop is generated against training requirements and training evidence, while an evaluation script is generated against metric and artifact requirements. Each generated file returns a provenance record linking file regions back to the requirement and evidence IDs they realize, and lists implementation choices that are inferred rather than directly specified by the contract. Validation produces two typed reports. Requirement review evaluates each active unit against its file contract and provenance record, and returns a verdict in Y. Runtime validation runs dependency installation, import checks, declared entry points, smoke commands, and benchmark commands when available. The two reports surface different failures: runtime validation catches broken code and missing artifacts, while requirement review catches runnable but paper-inconsistent implementations. When either report fails, ReproAgent issues a repair ticket rather than regenerating the repository. The ticket carries the failed units, target files, attached evidence, the edits required, and the units that already passed and must remain protected. Repair is file-local unless the ticket identifies an interface-level failure that requires coordinating two or more files, and repository regions whose contracts are unchanged skip redundant validation. The repair loop is bounded by a fixed budget; we use the same budget within each backbone setting and report the setting-specific value in §4.3. 4 Experimental Setup 4.1 Benchmark and Backbones We evaluate on PaperBench Code-Dev (Starace et al., 2025), which contains 20 ICML 2024 papers and hierarchical rubrics for repository-level code reproduction; the rubrics grade whether a generated repository implements paper-specific methods, protocols, and artifacts rather than only running. On this benchmark, the evaluation has three layers. The full-suite layer compares the primary ReproAgent run with reported PaperBench Code-Dev systems, using Claude-Sonnet-4.522 2 https://platform.claude.com/docs/en/about-claude/models/model-ids-and-versions. as the generation backbone. The same-backbone layer compares ReproAgent with simpler scaffolds under Gemini-3-Flash33 3 https://ai.google.dev/gemini-api/docs/gemini-3., the backbone shared with BasicAgent, IterAgent and AiScientist. The ablation layer removes each contract channel under the same Gemini-3-Flash backbone. 4.2 Comparisons and Metrics External rows reuse the model, scaffold, and runtime budget reported by their source. The AiScientist engineering report provides the BasicAgent, IterAgent, and AiScientist scaffold rows used in the same-backbone comparison. The aggregate full-suite figure includes only Code-Dev rows, and the per-paper table expands the main comparison to BasicAgent, IterAgent, AiScientist, DeepCode, and both ReproAgent backbones; Appendix B lists the additional external rows. We follow the Code-Dev judging protocol and report macro-averaged PaperBench score as the primary metric: weighted leaf scores aggregate to a per-paper score, and the 20 paper scores are averaged. Token, time, cost, and judge-provenance diagnostics are reported in the appendix. 4.3 Implementation and Ablations ReproAgent is implemented as a four-stage pipeline: the paper is chunked by section, reference repositories come from PaperBench metadata and related-code search subject to the benchmark blacklist on official target repositories, and planning freezes a global contract before file-level generation. Repair combines requirement review with sandboxed runtime validation, so failures become targeted tickets that patch only affected files. We cap stage-review repair at three attempts and repository-level repair at five rounds, uniformly across the Claude-Sonnet-4.5 main run and all Gemini-3-Flash full and ablation runs; holding this budget fixed isolates the channel effect from repair-budget effects. In total we report 20 papers × 4 ReproAgent settings = 80 reproduction runs, one run per (paper, setting) pair. Channel ablations. On the same Gemini-3-Flash backbone, No Implementation Requirements skips requirement extraction and requirement review, so architecture planning falls back on the paper’s section outline and reference-derived module structure; No Reference Evidence skips reference-repository surveying, so generation relies on requirement-derived contracts without bound content or structure evidence. All other components and repair budgets are held fixed. 5 Results and Analysis Figure 3: Full-suite PaperBench Code-Dev comparison across reported paper-to-code systems. Paper BasicAgent IterAgent DeepCode ReproAgent (ours) Adaptive Pruning 16.6 22.4↑ 5.8 54.4↑ 37.8 67.8↑ 51.2 All in One 24.8 11.5↓ 13.3 75.9↑ 51.1 76.9↑ 52.1 BAM 25.2 18.7↓ 6.5 74.8↑ 49.6 61.8↑ 36.6 BBOX 10.7 15.2↑ 4.5 64.4↑ 53.7 86.4↑ 75.7 Bridging Data Gaps 15.8 8.1↓ 7.7 58.1↑ 42.3 66.0↑ 50.2 FRE 22.7 17.5↓ 5.2 81.4↑ 58.7 76.8↑ 54.1 FTRL 9.3 4.3↓ 5.0 59.8↑ 50.5 63.9↑ 54.6 LBCS 19.8 20.7↑ 0.9 74.7↑ 54.9 65.8↑ 46.0 LCA on the Line 14.0 13.3↓ 0.7 74.9↑ 60.9 64.9↑ 50.9 Mechanistic Understanding 18.1 3.7↓ 14.4 92.5↑ 74.4 75.7↑ 57.6 PINN 24.8 22.5↓ 2.3 91.0↑ 66.2 81.5↑ 56.7 RICE 19.8 7.8↓ 12.0 70.2↑ 50.4 75.1↑ 55.3 Robust CLIP 29.6 24.7↓ 4.9 73.3↑ 43.7 63.0↑ 33.4 Sample-specific Masks 30.9 25.7↓ 5.2 67.1↑ 36.2 82.0↑ 51.1 SAPG 5.4 6.3↑ 0.9 73.8↑ 68.4 86.5↑ 81.1 Sequential Neural Score Estimation 41.4 32.7↓ 8.7 87.0↑ 45.6 78.0↑ 36.6 Stay on Topic with Classifier-Free Guidance 10.8 16.2↑ 5.4 70.5↑ 59.7 58.5↑ 47.7 Stochastic Interpolants 12.4 26.3↑ 13.9 81.5↑ 69.1 80.8↑ 68.4 Test-Time Model Adaptation 14.3 14.2↓ 0.1 64.9↑ 50.6 75.6↑ 61.3 What Will My Model Forget 29.7 11.4↓ 18.3 80.8↑ 51.1 87.6↑ 57.9 Mean score 19.8 16.2↓ 3.6 73.5↑ 53.7 73.7↑ 53.9 Table 1: Per-paper PaperBench Code-Dev comparison with Claude backbones. Claude columns for BasicAgent and IterAgent follow the PaperBench report (Starace et al., 2025); DeepCode follows its concurrent report (Li et al., 2025b). BasicAgent and IterAgent use Claude-3.5-Sonnet, while DeepCode and ReproAgent (ours) use Claude-4.5-Sonnet. Subscripts show absolute-point differences from BasicAgent on the same paper; red upward and green downward arrows indicate the sign of the difference. best and second-best mark the row-best and row-second-best scores among the shown Claude columns. Paper BasicAgent IterAgent AiScientist ReproAgent (ours) Adaptive Pruning 24.5 3.0↓ 21.5 27.2↑ 2.7 41.9↑ 17.4 All in One 20.9 45.1↑ 24.2 46.3↑ 25.4 51.6↑ 30.7 BAM 48.5 45.0↓ 3.5 56.6↑ 8.1 48.5 BBOX 15.4 8.3↓ 7.1 33.8↑ 18.4 15.6↑ 0.2 Bridging Data Gaps 12.6 12.4↓ 0.2 23.1↑ 10.5 50.6↑ 38.0 FRE 21.7 23.9↑ 2.2 35.2↑ 13.5 26.9↑ 5.2 FTRL 5.9 4.2↓ 1.7 10.1↑ 4.2 40.1↑ 34.2 LBCS 17.8 15.3↓ 2.5 27.9↑ 10.1 42.5↑ 24.7 LCA on the Line 13.0 18.3↑ 5.3 30.2↑ 17.2 41.8↑ 28.8 Mechanistic Understanding 14.9 21.9↑ 7.0 29.9↑ 15.0 48.1↑ 33.2 PINN 26.6 30.8↑ 4.2 49.9↑ 23.3 61.3↑ 34.7 RICE 10.4 8.9↓ 1.5 10.9↑ 0.5 31.0↑ 20.6 Robust CLIP 15.4 10.4↓ 5.0 18.3↑ 2.9 29.5↑ 14.1 Sample-specific Masks 25.4 33.3↑ 7.9 36.8↑ 11.4 53.8↑ 28.4 SAPG 11.4 12.7↑ 1.3 19.9↑ 8.5 29.0↑ 17.6 Sequential Neural Score Estimation 53.5 60.2↑ 6.7 64.9↑ 11.4 45.7↓ 7.8 Stay on Topic with Classifier-Free Guidance 8.4 13.7↑ 5.3 20.1↑ 11.7 25.3↑ 16.9 Stochastic Interpolants 17.0 17.4↑ 0.4 18.8↑ 1.8 39.1↑ 22.1 Test-Time Model Adaptation 15.3 18.1↑ 2.8 32.5↑ 17.2 54.9↑ 39.6 What Will My Model Forget 6.6 9.0↑ 2.4 17.9↑ 11.3 16.8↑ 10.2 Mean score 19.3 20.6↑ 1.3 30.5↑ 11.2 39.7↑ 20.4 Table 2: Per-paper PaperBench Code-Dev comparison with Gemini. Gemini columns for BasicAgent, IterAgent, and AiScientist follow the AiScientist engineering report (Chen et al., 2026); all Gemini columns use Gemini-3-Flash. Subscripts show absolute-point differences from BasicAgent on the same paper; red upward and green downward arrows indicate the sign of the difference. best and second-best mark the row-best and row-second-best scores among the shown Gemini columns. 5.1 Full-Suite Comparison Figure 3 reports mean PaperBench Code-Dev scores across published paper-to-code systems. ReproAgent with Claude-Sonnet-4.5 reaches 73.7, above PaperCoder (Seo et al., 2026) (45.1), AutoP2C (Lin et al., 2025) (49.2), AutoReproduce (Zhao et al., 2025) (49.6), and Deep-Reproducer (Chen et al., 2025) (63.2), and edges past the concurrent state-of-the-art DeepCode at 73.5. Tables 1 and 2 expand this view into the 20 targets for each backbone; additional external rows appear in Appendix B. We hypothesize that the gain comes from the contract itself: requirement and evidence bindings that survive across files keep dependent artifacts, such as method, training, and evaluation code, consistent during generation and repair instead of being re-derived per stage. 5.2 Same-Backbone Comparison Holding the backbone fixed at Gemini-3-Flash, ReproAgent reaches 39.7 against 19.3 for BasicAgent, 20.6 for IterAgent, and 30.5 for AiScientist. The advantage over the strongest external Gemini row is on the order of 9 absolute points, consistent with the contract effect we hypothesize in the full-suite comparison. 5.3 Ablation Study Figure 4: Gemini channel ablations on all 20 papers. Removing reference evidence (top) or implementation requirements (bottom) lowers the mean; judge provenance and token diagnostics are in the appendix. Setting Mean Median Δ Full contract 39.7 41.8 — −- Reference evidence 21.6 21.4 −-18.1 −- Implementation requirements 25.6 24.6 −-14.1 Table 3: Gemini-3-Flash ablation results over 20 papers; Δ is the mean drop from the full contract. We next isolate the contribution of each contract channel under Gemini-3-Flash. Table 3 summarizes the aggregates: removing reference evidence drops the mean by 18.1 and removing implementation requirements drops it by 14.1. The full setting beats both ablations on every one of the 20 targets, so each channel contributes capability that the other cannot recover. Requirement channel: Sample-specific Masks. Sample-specific Masks scores 53.8 with the full contract, 21.3 without implementation requirements, and 28.0 without reference evidence: the requirement drop is the dominant one. The paper’s contribution is a small set of named, implementation-critical obligations, including a per-sample mask generator, interpolation from a low-resolution mask up to the input image size, joint optimization of a shared pattern and the mask-generator parameters, the PAD/NARROW/MEDIUM/FULL baselines, and ablations over shared-pattern and mask channels. The full contract pins these obligations to specific work packages and files, so the generated repository keeps the baseline and ablation method registries that the rubric checks. Without the requirement channel, the planner falls back on the section outline and reference-derived modules, and the generated repository keeps runnable scaffolding but drops the named baselines, ablations, and mask-channel switches, since none of them are pinned to a file. The case-level code excerpt and a per-paper score table for additional cases appear in Appendix C. Evidence channel: Bridging Data Gaps. Bridging Data Gaps scores 50.6 with the full contract, 38.4 without implementation requirements, and 13.9 without reference evidence: the evidence drop is the dominant one. The paper names the high-level task and adaptor objectives, but its concrete protocol surface, including file layout, training-loop wiring, checkpoint and trace artifacts, and configuration handling, is supplied by reference repositories rather than the paper text. With the full contract, the generated repository emits the expected checkpoint, training-trace, and resolved-configuration artifacts under conventional output paths, matching the pattern of checkpointed training and reproducible configuration logging. Without the evidence channel, paper obligations remain visible in the plan, but the generated repository under-implements file layout, data-handling conventions, experiment wiring, and artifact naming, so it fails to produce these expected artifacts. Appendix C gives the corresponding code-level trace and Appendix Table 6 lists the per-paper scores behind these aggregates. 5.4 Discussion Two observations stand out. First, the same-backbone advantage and the full-suite gain move together, so the lift attaches to the contract rather than to any single backbone. Second, the two channels are complementary: removing either drops the mean by more than 14 points, and the failures fall into two classes, paper-specific obligations dropped without the requirement channel and repository-level scaffolding lost without the evidence channel. Together, these support a mechanistic claim: ReproAgent benefits from a global contract whose requirement channel preserves explicit paper obligations and whose evidence channel grounds implicit repository knowledge. 6 Conclusion We presented ReproAgent, a contract-guided pipeline for paper-to-code reproduction. Its global contract binds explicit paper obligations and implicit reference-repository evidence to work packages, file contracts, and repair decisions. On PaperBench Code-Dev, ReproAgent scores 73.7 across all 20 papers; same-backbone rows support scaffold-level gains, while channel ablations and case traces support the role of both contract channels. The takeaway is that paper obligations and reference-repository evidence are most useful when persisted as a contract: once both are bound, the same artifact anchors planning, exposes coverage failures, and scopes repair. 7 Limitations ReproAgent targets repository-level reproduction in which the paper and benchmark-allowed reference material jointly specify the implementation target. Extending the approach to other reproduction settings, modalities, and evaluation protocols is a natural direction for future work. We report token, time, and call diagnostics in Appendix G to make the process auditable rather than to compare deployment cost; further engineering optimizations are compatible with the contract design and are also left to future work. References Baek et al. (2024) J. Baek, S. S. Jauhar, S. Cucerzan, and S. J. Hwang ResearchAgent: iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738. External Links: Link Cited by: §2. Baker (2016) M. Baker 1,500 scientists lift the lid on reproducibility. Nature 533 (7604), p. 452–454. External Links: Document, Link Cited by: §1. Baumgärtner and Gurevych (2026) T. Baumgärtner and I. Gurevych SciCoQA: quality assurance for scientific paper–code alignment. arXiv preprint arXiv:2601.12910. External Links: Link Cited by: §1, §2. Belz et al. (2021) A. Belz, S. Agarwal, A. Shimorina, and E. Reiter A systematic review of reproducibility research in natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), p. 381–393. External Links: Document, Link Cited by: §1. Chan et al. (2024) J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. Mądry MLE-bench: evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095. External Links: Link Cited by: §2. Chen et al. (2026) G. Chen, J. Chen, L. Chen, J. Zhao, F. Meng, W. X. Zhao, R. Song, C. Chen, J. Wen, and K. Jia Toward autonomous long-horizon engineering for ML research. arXiv preprint arXiv:2604.13018. External Links: Link Cited by: Table 4, Table 4, Table 4, Table 2. Chen et al. (2025) P. Chen, N. Yan, Z. Zhao, Y. Lin, H. Chen, Y. Hu, Q. Bai, X. Li, and M. S. Mortazavi Deep-Reproducer: from paper understanding to code generation. In NeurIPS 2025 Fourth Workshop on Deep Learning for Code, External Links: Link Cited by: Table 4, §5.1. Chen et al. (2024) T. Chen, H. Wang, S. Chen, W. Yu, K. Ma, X. Zhao, H. Zhang, and D. Yu Dense X retrieval: what retrieval granularity should we use?. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, External Links: Document, Link Cited by: §2. Cheng et al. (2024) W. Cheng, Y. Wu, and W. Hu Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 7957–7977. External Links: Document, Link Cited by: §2. Cleland-Huang et al. (2014) J. Cleland-Huang, O. C. Z. Gotel, J. Huffman Hayes, P. Mäder, and A. Zisman Software traceability: trends and future directions. In Proceedings of the Future of Software Engineering (FOSE), p. 55–69. External Links: Document, Link Cited by: §2. Gotel and Finkelstein (1994) O. C. Z. Gotel and A. C. W. Finkelstein An analysis of the requirements traceability problem. In Proceedings of the IEEE International Conference on Requirements Engineering (ICRE), p. 94–101. External Links: Document, Link Cited by: §2. Hong et al. (2026) H. Hong, Y. Li, J. Chen, S. Ananiadou, X. Li, J. Kim, and C. Lin HiRAS: a hierarchical multi-agent framework for paper-to-code generation and execution. arXiv preprint arXiv:2604.17745. External Links: Link Cited by: §2. Hong et al. (2024) S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber MetaGPT: meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2. Islam et al. (2024) Md. A. Islam, M. E. Ali, and M. R. Parvez MapCoder: multi-agent code generation mirroring human cognitive processes. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Document, Link Cited by: §2. Li et al. (2025a) J. Li, X. Shi, K. Zhang, G. Li, Z. Jin, L. Li, H. Zhang, F. Liu, Y. Zhang, Z. Tao, Y. Dong, Y. Zhu, and C. Tao GraphCodeAgent: dual graph-guided LLM agent for retrieval-augmented repo-level code generation. arXiv preprint arXiv:2504.10046. External Links: Link Cited by: §2. Li et al. (2025b) Z. Li, Z. Li, Z. Guo, X. Ren, and C. Huang DeepCode: open agentic coding. arXiv preprint arXiv:2512.07921. External Links: Link Cited by: Table 4, §2, Table 1. Lin et al. (2025) Z. Lin, Y. Shen, Q. Cai, H. Sun, J. Zhou, and M. Xiao AutoP2C: an LLM-based agent framework for code repository generation from multimodal content in academic papers. arXiv preprint arXiv:2504.20115. External Links: Link Cited by: Table 4, §2, §5.1. Lu et al. (2024) C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha The AI scientist: towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292. External Links: Link Cited by: §2. Meyer (1992) B. Meyer Applying “design by contract”. Computer 25 (10), p. 40–51. External Links: Document, Link Cited by: §2. Pimparkhede et al. (2024) S. Pimparkhede, S. Kammakomati, S. Tamilselvam, P. Kumar, A. Bhatt, and P. Kumar DocCGen: document-based controlled code generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, External Links: Document, Link Cited by: §2. Pineau et al. (2021) J. Pineau, P. Vincent-Lamarre, K. Sinha, V. Larivière, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and H. Larochelle Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program). Journal of Machine Learning Research 22 (164), p. 1–20. External Links: Link Cited by: §1. Qian et al. (2024) C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun ChatDev: communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 15174–15186. External Links: Document, Link Cited by: §2. Schmidgall et al. (2025) S. Schmidgall, Y. Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, Z. Liu, and E. Barsoum Agent laboratory: using LLM agents as research assistants. arXiv preprint arXiv:2501.04227. External Links: Link Cited by: §2. Seo et al. (2026) M. Seo, J. Baek, S. Lee, and S. J. Hwang Paper2Code: automating code generation from scientific papers in machine learning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Table 4, §1, §2, §5.1. Sethi et al. (2018) A. Sethi, A. Sankaran, N. Panwar, S. Khare, and S. Mani DLPaper2Code: auto-generation of code from deep learning research papers. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, p. 899. External Links: Document, Link Cited by: §2. Siegel et al. (2024) Z. S. Siegel, S. Kapoor, N. Nagdir, B. Stroebl, and A. Narayanan CORE-Bench: fostering the credibility of published research through a computational reproducibility agent benchmark. Transactions on Machine Learning Research (TMLR). External Links: Link Cited by: §2. Starace et al. (2025) G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, J. Heidecke, A. Glaese, and T. Patwardhan PaperBench: evaluating AI’s ability to replicate AI research. In Proceedings of the International Conference on Machine Learning (ICML), External Links: Link Cited by: Table 4, Table 4, §1, §2, §4.1, Table 1, Abstract. Tang et al. (2024) X. Tang, Y. Liu, Z. Cai, Y. Shao, J. Lu, Y. Zhang, Z. Deng, H. Hu, K. An, R. Huang, S. Si, S. Chen, H. Zhao, L. Chen, Y. Wang, T. Liu, Z. Jiang, B. Chang, Y. Fang, Y. Qin, W. Zhou, Y. Zhao, A. Cohan, and M. Gerstein ML-Bench: evaluating large language models and agents for machine learning tasks on repository-level code. External Links: 2311.09835, Link Cited by: §2. Xia et al. (2025) C. S. Xia, Y. Deng, S. Dunn, and L. Zhang Demystifying LLM-based software engineering agents. Proceedings of the ACM on Software Engineering 2 (FSE). External Links: Document, Link Cited by: §2. Xiang et al. (2025) Y. Xiang, H. Yan, S. Ouyang, L. Gui, and Y. He SciReplicate-Bench: benchmarking LLMs in agent-driven algorithmic reproduction from research papers. In Proceedings of the Conference on Language Modeling (COLM), Note: arXiv:2504.00255. External Links: Link Cited by: §1, §2. Yan et al. (2025) S. Yan, R. Li, Z. Luo, Z. Wang, D. Li, L. Jing, K. He, P. Wu, J. Ni, G. Michalopoulos, Y. Zhang, Z. Zhang, M. Zhang, Z. Chen, and X. Du LMR-BENCH: evaluating LLM agent’s ability on reproducing language modeling research. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, p. 6164–6186. External Links: Document, Link Cited by: §2. Yang et al. (2024) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press SWE-agent: agent-computer interfaces enable automated software engineering. In Proceedings of the 38th International Conference on Neural Information Processing Systems, External Links: Link Cited by: §2. Zhang et al. (2023) F. Zhang, B. Chen, Y. Zhang, J. Keung, J. Liu, D. Zan, Y. Mao, J. Chen, and W. Chen RepoCoder: repository-level code completion through iterative retrieval and generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Document, Link Cited by: §2. Zhang et al. (2024) Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury AutoCodeRover: autonomous program improvement. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), External Links: Document, Link Cited by: §2. Zhao et al. (2025) X. Zhao, Z. Sang, Y. Li, Q. Shi, W. Zhao, S. Wang, D. Zhang, X. Han, Z. Liu, and M. Sun AutoReproduce: automatic AI experiment reproduction with paper lineage. arXiv preprint arXiv:2505.20662. External Links: Link Cited by: Table 4, §1, §2, §5.1. Zhou et al. (2025) M. Zhou, Q. Yao, L. Du, L. Wei, and D. Zheng Reflective paper-to-code reproduction enabled by fine-grained verification. arXiv preprint arXiv:2508.16671. External Links: Link Cited by: Table 4, §1, §2. Zhou et al. (2023) S. Zhou, U. Alon, F. F. Xu, Z. Wang, Z. Jiang, and G. Neubig DocPrompting: generating code by retrieving the docs. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2. Appendix A Contract Case Study: Unit and Evidence Records This appendix gives two paired schema records for a continual-RL paper that introduces a forgetting-aware fine-tuning protocol. The first illustrates the implementation-requirement schema in §3.1; the second illustrates the reference-evidence schema in §3.2 and is anchored to the unit from the first. The records are compact schema views that preserve source anchors and evidence bindings used by review and repair. A.1 An Implementation-Requirement Unit The unit below illustrates how Prepare extracts a requirement, Plan assigns it to a method work package, Generate projects it into a model-method file contract, and Repair verifies it with requirement review. ⬇ "unit_id": "unit_004", "type": "method", "statement": "implement Elastic Weight Consolidation (EWC)", "paper_evidence": [ "L_EWC = L_RL + (lambda/2) * sum_i F_i (theta_i - theta_*_i)^2", "where theta_* is the pre-trained model and F is the diagonal of the Fisher matrix." ], "source_paragraph_ids": ["chunk_004_02", "chunk_024"], "verification_targets": [ "kind": "artifact", "description": "Fisher matrix calculation and EWC loss implementation" ], "implementation_surfaces": ["model_or_method"], "code_obligations": [ "Implement diagonal Fisher Information Matrix from the pre-trained policy on its training distribution.", "Implement the EWC loss term (lambda/2) * sum_i F_i (theta_i - theta_*_i)^2.", "Use lambda=10000 as the default coefficient." ], "status": "active" What each field stores. • unit_id is a stable identifier; repair tickets and provenance records both index by this value, so the same unit remains addressable across stages. • type categorizes the unit (here: method) and drives how Plan groups it with related units into a work package. • statement is the normalized one-duty implementation requirement, derived from the paper rather than copied verbatim. • paper_evidence keeps the originating paper quotes, so review can cross-check the statement against the source. • source_paragraph_ids are anchors into the segmented paper (chunk IDs), used to locate the unit in the addendum or main body and to guide review and repair. • verification_targets declares what reviewers should check; each target has a kind (artifact, surface, or metric) and a description. • implementation_surfaces declares where the obligation should appear in code; file planning must project the unit onto at least one file whose declared surfaces cover this set. • code_obligations expand the statement into concrete sub-duties that generation and review can verify pointwise. • status is the lifecycle flag; review writes verdicts (pass, missing, wrong-surface, inconsistent-with-source) back into this field. How the preservation invariant fires on this unit. Plan cannot proceed unless the EWC unit is bound to at least one owner work package; in this run it is owned by a method package together with the related PPO-fine-tuning unit. File planning cannot proceed unless the unit is projected into at least one file contract whose declared surface includes model_or_method; in this run the unit lands in a single dedicated method file. Generate cannot finalize that file unless the file emits a provenance record citing unit_004, so the unit id surfaces in the per-file provenance map alongside its source paragraphs. Repair cannot mark the unit pass until requirement review verifies all three code obligations against the file contract; otherwise it issues a repair ticket carrying the unit id, the matched evidence ids attached to the work package, and the protected units that already passed. The unit is therefore not a free-form paragraph or a planning-time prompt fragment. It is an addressable record with a fixed schema that planning, file projection, generation, validation, and repair are all conditioned on, which is what allows the same paper requirement to remain visible across the four-stage pipeline rather than being silently dropped. A.2 A Reference-Evidence Item The evidence item below illustrates how a selected reference snippet is scoped to the method work package that owns the EWC unit unit_004. ⬇ "work_package_id": "wp_methods", "unit_id": "unit_004", "ref_id": "ref_001", "file_path": "model.py", "snippet_preview": "model.py:13: def __init__(self, in_features, out_features, sigma0=0.5): .py:14: super().__init__() .py:15: self.in_features = in_features .py:16: self.out_features = out_features .py:17: self.weight = n.Parameter(torch.Tensor(out_features, in_features)) .py:18: self.bias = n.Parameter(torch.Tensor(out_features)) .py:19: …", "why_relevant": "function ‘__init__‘ in ‘model.py‘ aligns with units paper_method_core, unit_004 via parameter construction patterns reusable for the EWC reference network.", "confidence": 1.0, "matched_keywords": ["n.Parameter", "in_features", "out_features"] What each field stores. • work_package_id names the package this evidence is attached to; in §3.2 the same identifier serves as the binding point between requirements and evidence. • unit_id names the supported requirement; here it is the EWC unit unit_004 from Appendix A.1, which is what makes the evidence consumable by the same package contract. • ref_id, file_path, and snippet_preview together pin down (repo,path,region)(repo,path,region): the source repository, the file inside it, and the matched line range, recorded so review and repair can locate the snippet again. • why_relevant is the structured match basis: it explains, in one sentence, which symbol or interface aligns the snippet with the supported unit, rather than leaving the LLM judgment as a free-form note. • confidence and matched_keywords record the auxiliary signals used by the retain rule in §3.2, namely a symbol-level match strengthened by an LLM judgment. • The role field, omitted from this stripped record but tracked elsewhere in the run state, marks this snippet as reusable structural evidence for parameter construction. How the evidence enters generation. The item is bound to wp_methods during Plan, so when Generate writes the file contract that owns unit_004, the prompt for that file sees this snippet but no unrelated reference code from sibling work packages. If requirement review later returns a non-pass verdict on unit_004, the repair ticket carries the same ref_id and file_path, allowing repair to re-anchor the edit on this specific reference snippet rather than searching the reference repository again. Appendix B External Baselines and Cost Notes Table 4 records aggregate external rows that are useful context but not part of the primary same-backbone comparison. Rows include official PaperBench results, recent paper-to-code systems, and separately collected external reference rows. They differ in scaffold, model, runtime budget, and reporting protocol, and provide contextual reported scores rather than a primary controlled comparison. Method Backbone / setting Reported score BasicAgent (Starace et al., 2025)a Gemini-2.0-Flash 5.0 o3-mini 5.1 GPT-4o 7.7 o1 19.5 Claude-3.5-Sonnet 35.4 IterativeAgent (Starace et al., 2025)a o3-mini 16.4 Claude-3.5-Sonnet 27.5 o1 43.3 PaperCoder (Seo et al., 2026) o3-mini-high 45.1 AutoP2C (Lin et al., 2025) source-reported setting 49.2 AutoReproduce (Zhao et al., 2025) source-reported setting 49.6 Deep-Reproducer (Chen et al., 2025) GPT-5 63.2 RePro (Zhou et al., 2025)b o3-mini 62.6 DeepCode (Li et al., 2025b)c Claude-Sonnet-4.5 73.5 BasicAgent (Chen et al., 2026)d Gemini-3-Flash 19.3 IterAgent (Chen et al., 2026)d Gemini-3-Flash 20.6 AiScientist (Chen et al., 2026)d Gemini-3-Flash 30.5 ReproAgent (ours) Claude-Sonnet-4.5 73.7 Gemini-3-Flash 39.7 Table 4: Supplemental aggregate PaperBench baselines. Reported scores are percentages. Rows differ in scaffold, backbone, runtime budget, and reporting protocol, and provide context for the full-suite comparison rather than a primary controlled comparison. a Official 20-paper PaperBench Code-Dev scaffold rows; standard errors are reported in the cited source, and this table reports means only for compact contextual comparison. The o1 IterativeAgent row uses two seeds. b RePro reports PRroot@5, so we list it as external PRroot context separate from the Code-Dev rows. c Collected 20-paper mean over three runs per paper; the reported mean standard error across papers is 2.8 and the mean cost is $8.88. d Collected AiScientist engineering aggregates; mean costs are $6.25, $27.44, and $15.67 for BasicAgent, IterAgent, and AiScientist. Appendix C Gemini Ablation Case Studies We keep qualitative ablation cases in the appendix so that the main results focus on aggregate behavior. The examples below illustrate the channel roles behind the aggregate ablation pattern in Section 5.3. Scores in this section provide mechanism-oriented evidence for the aggregate analysis. C.1 Tracing One Unit in Test-Time Model Adaptation We trace Test-Time Model Adaptation because its FOA/CMA unit is a strong trace for the implementation-requirement channel and a useful contrast for how reference evidence affects repository organization. In our consolidated 2026-05-24 PaperBench judging snapshot, the full Gemini run scores 54.9, while removing implementation requirements scores 47.1 and removing reference evidence scores 23.8. The trace is taken from the corresponding Gemini full-run artifacts for test-time-model-adaptation. Extraction. unit_002 is extracted as “Implement Forward-Only Prompt Adaptation with CMA”. Its obligations require prompt insertion into the ViT input sequence, a CMA-ES optimizer, a population size K∈[2,28]K∈[2,28], and a forward-only prompt update loop. unit_003 is extracted as the activation-distribution discrepancy fitness function, requiring mean/variance statistics for test CLS tokens and a fitness value passed to CMA. Work package and contract. Planning binds both units to wp_foa_core, whose goal is to implement the FOA method and CMA optimizer. The work package produces methods/foa.py, optimizers/cma_es.py, and results/source_stats.json. Its global contract preserves the FOA loop, CMA-ES interface, source-statistics collector, fitness function, method registry, and parameter inventory for K, prompt length, alignment weights, batch size 64, and momentum 0.9. This is the implementation-requirement channel in concrete form: the paper obligations become file-owned duties rather than prose context. Evidence binding. The evidence bundle for wp_foa_core records paper chunks for FOA, source-statistics calculation, and the evaluation protocols. In this case, the package is marked self_contained and has no repository evidence_links, because the decisive FOA/CMA obligations are already present in the paper and addendum evidence. This clarifies the ablation: the no-reference setting still recovers much of the requirement surface, while the full setting adds stronger repository-level organization, protocol wiring, and comparison surfaces. Generated files and provenance. Generation materializes the contract in three files. methods/foa.py contains the hyperparameter sweeps, method/baseline registries, a CMA-ES implementation, source-statistics loading, and FOA routines. optimizers/cma_es.py exposes a standalone CMA-ES ask/tell optimizer and method selectors. src/foa/utils/cma_optimizer.py contains a forward-only FOA.adapt route under torch.no_grad(). The generation file-provenance record assigns all three files to wp_foa_core, lists unit_002 and unit_003 as owned units, and attaches validation checks for implementation requirements, formula/algorithm contracts, evidence-matrix coverage, and route closure. The key point is that the requirement IDs, work-package owner, generated files, and validation checks remain connected throughout the trajectory; the ablation removes one of these channels before the file-level contract is formed. C.2 Tracing Reference Evidence in Bridging Data Gaps Bridging Data Gaps is a complementary case because the same snapshot shows an asymmetric ablation: the full Gemini run scores 50.6, the run without implementation requirements scores 38.4, and the run without reference evidence scores 13.9. The case therefore stresses the second channel in Figure 2: the paper requirement channel keeps the high-level adaptation and artifact-writing obligations visible, while reference evidence helps convert those obligations into a concrete repository protocol. Requirement channel. The requirement units preserve the paper-derived duties that must survive into code: train the adaptation components, expose a runnable experiment path, save model/adaptor checkpoints, and write result artifacts that make the experiment inspectable. These duties are projected into file contracts that expect a training route and artifact writer rather than a single monolithic script. This explains the asymmetric score pattern: the remaining reference and generic task context can still support some repository structure, but the paper-specific obligations are less explicit without the requirement channel. Reference-evidence channel. The reference channel supplies the implementation texture that the paper does not spell out line by line. In the full run, the generated repository writes checkpoints/adaptor.pth, checkpoints/trained_model.pth, results/ant_training_trace.json, and results/config_resolved.json, matching the expected pattern of checkpointed training and reproducible configuration logging. When reference evidence is removed, the paper obligations remain visible, but the run has less support for file layout, data-handling conventions, experiment wiring, and artifact naming. This case provides a concrete trace of why the reference-evidence channel can matter even when the target paper already names the high-level task. C.3 Additional Ablation Patterns The ablation pattern is not driven by a single paper. The excerpts below are illustrative traces from the full Gemini repositories, not additional scoring criteria. They show how the contract is materialized as file-owned implementation choices rather than remaining only in planning prose. ⬇ BASELINES = ["PAD", "NARROW", "MEDIUM", "FULL"] ABLATIONS = ["ONLY_delta", "ONLY_f_mask", "SINGLE_CHANNEL_f_mask_s", "OURS"] def get_method_config(method_name: str) -> MethodConfig: if method_name == "ONLY_delta": return MethodConfig(name="ONLY_delta", use_delta=True, use_f_mask=False) elif method_name == "ONLY_f_mask": return MethodConfig(name="ONLY_f_mask", use_delta=False, use_f_mask=True) elif method_name in ["ours", "OURS", "SMM"]: return MethodConfig(name="OURS", use_delta=True, use_f_mask=True, mask_channels=3) elif method_name in ["PAD", "NARROW", "MEDIUM", "FULL"]: return MethodConfig(name=method_name, use_delta=True, use_f_mask=False) ⬇ for step in range(num_iterations): loss_val = train_ant_step(batch, config, model, adaptor, classifier, optimizer) trace.append("step": step, "loss": loss_val) os.makedirs("checkpoints", exist_ok=True) os.makedirs("results", exist_ok=True) torch.save(adaptor.state_dict(), "checkpoints/adaptor.pth") torch.save(model.state_dict(), "checkpoints/trained_model.pth") with open("results/ant_training_trace.json", "w") as f: json.dump(trace, f, indent=2) with open("results/config_resolved.json", "w") as f: json.dump(config, f, indent=2) Paper Full w/o impl. w/o ref. Case role Sample-specific Masks 53.8 21.3 28.0 Clear two-channel benefit: the full run preserves SMM-specific optimization, mask generation, interpolation, baselines, and ablations. BAM 48.5 30.8 20.7 Large positive case: both removed-channel settings score substantially below the full contract, with the largest gap for reference evidence. Bridging Data Gaps 50.6 38.4 13.9 Reference-heavy case: paper obligations remain useful, but removing reference evidence sharply reduces concrete protocol and repository-structure recovery. PINN 61.3 53.3 52.2 Standardized-prior case: all variants remain strong because PINN objectives and PDE-loss structure are common, but the full contract is still best. What Will My Model Forget 16.8 10.7 8.4 Scale and coverage stress case: the task has many long-tail details, and the full contract still leads both ablations. LCA-on-the-Line 41.8 16.1 33.8 Boundary case: no-reference keeps a visible checklist for ObjectNet/LCA details, while the full run remains the strongest setting. Table 5: Representative Gemini ablation cases. The case role column summarizes inspection observations alongside the score columns. “w/o impl.” removes the implementation-requirement channel; “w/o ref.” removes the reference-evidence channel. Figure 5: Code-level evidence for two representative ablation cases. The full generated repositories expose different concrete surfaces: Sample-specific Masks relies on implementation-requirement preservation for method-specific masks, interpolation, baselines, and ablations, while Bridging Data Gaps relies more heavily on reference evidence for protocol and training-artifact wiring. Large two-channel gains. Sample-specific Masks and BAM are representative qualitative traces covering large two-channel gains. In Sample-specific Masks, the full run keeps the paper’s required surfaces connected: the mask generator, interpolation from a low-resolution mask to the input image size, joint optimization of the shared pattern and mask-generator parameters, PAD/NARROW/MEDIUM/FULL baselines, and ablations over shared pattern and mask channels. Removing implementation requirements disrupts this preservation path, so the repository retains runnable scaffolding but covers fewer paper-specific obligations. Removing reference evidence keeps the obligations but reduces the file and experiment organization needed to expose baselines and ablation routes in a concrete implementation. BAM shows a similar pattern at a coarser level: the full score is 48.5, while no-implementation and no-reference fall to 30.8 and 20.7, respectively. This indicates that the reference-evidence channel recovers practical implementation and evaluation structure, while the implementation-requirement channel keeps paper-specific modeling and protocol obligations explicit. Reference-sensitive cases. Bridging Data Gaps is useful because the two ablations degrade asymmetrically. The full run reaches 50.6, the no-implementation run remains at 38.4, and the no-reference run drops to 13.9. The artifact trace shows that the implementation-requirement channel preserves high-level tasks and expected outputs, while reference evidence supplies much of the concrete protocol surface: file layout, scripts, data handling conventions, and experiment wiring. This is the kind of paper where related repositories supply implementation texture that is not explicit enough in the paper text alone. Standardized tasks. PINN is a high-scoring but smaller-margin case. The full run obtains 61.3, while the two ablations remain at 53.3 and 52.2. This pattern is consistent with the channel hypothesis. PINN-style repositories have strong public priors: collocation points, PDE residual losses, boundary losses, optimizer loops, and diagnostic plots are common implementation patterns. As a result, Gemini can recover a reasonable repository even with one channel removed. The full contract still gives the best score because it binds these common priors to the paper-specific objective, PDE setup, loss-landscape diagnostics, and expected artifacts. Scale and coverage stress. What Will My Model Forget is a scale-sensitive case for the Gemini backbone. The full setting scores 16.8 and both ablations are lower. The target has broad long-tail coverage requirements and many benchmark-specific details, so this case separates channel utility from backbone capacity while preserving the same ordering pattern. C.4 Why Ablation Gaps Vary The per-paper ablation gaps are heterogeneous, and this variation is itself informative. For this discussion, we define the reference-evidence gain as the difference between the full Gemini score and the score without reference evidence, and the implementation-requirement gain as the difference between the full Gemini score and the score without implementation requirements. The largest reference-evidence gains occur on Bridging Data Gaps (+36.7), Adaptive Pruning (+35.7), Test-Time Adaptation (+31.1), FTRL (+28.4), All-in-One (+27.9), and BAM (+27.8). The largest implementation-requirement gains occur on Mechanistic Understanding (+43.5), Sample-Specific Masks (+32.5), Adaptive Pruning (+27.6), FTRL (+26.7), LCA-on-the-Line (+25.7), and All-in-One (+23.6). These rankings suggest that the two channels help for related but distinguishable reasons. Reference transferability. Reference evidence helps most when external artifacts provide transferable implementation structure: dataset protocols, baseline organization, metric wiring, experiment scripts, configuration conventions, or repository layout. This is broader than simply counting cloned repositories. For example, BBOX has multiple reference or dataset repositories in the preparation artifacts, yet its reference-evidence gain is small (+1.0), showing that repository count alone is not the key factor. Conversely, Bridging Data Gaps shows a large reference-evidence gain even with a compact prepared reference-repository list, because the useful signal is whether reference evidence can be fused into concrete protocol and repository-structure decisions. In reference-sensitive cases, removing this channel leaves the paper obligations visible but makes the generated repository less able to recover executable data handling, experiment wiring, and comparison surfaces. Requirement hardness. The implementation-requirement channel helps most when a paper’s contribution is compact but implementation-critical: a small set of named mechanisms, constraints, losses, ablations, metrics, or artifacts determines whether the repository is faithful. Sample-Specific Masks is a representative case: the decisive obligations include mask generation, interpolation from a low-resolution mask to image size, joint optimization of shared patterns and mask-generator parameters, PAD/NARROW/MEDIUM/FULL baselines, and ablations over shared-pattern and mask channels. Mechanistic Understanding, Adaptive Pruning, FTRL, and LCA-on-the-Line exhibit the same pattern at different scales: without explicit implementation requirements, the model can still produce plausible scaffolding, but it tends to omit or dilute the paper-specific mechanisms and protocol obligations that decide the score. Small-gap cases. Small ablation gaps do not necessarily imply that the removed channels are irrelevant. They usually fall into one of three categories. First, some papers have strong public implementation priors; PINN is the clearest example, where collocation points, PDE residual losses, boundary losses, optimizer loops, and diagnostic plots are common enough that both ablations remain strong. Second, some papers have compressed gaps because all variants face the same central implementation bottleneck; BBOX is a case where the three Gemini variants remain close. Third, some tasks are scale or coverage stress cases, such as What Will My Model Forget, where the full contract still leads but the target contains many long-tail requirements. Overall, the ablation gaps are best interpreted as indicators of where each channel provides leverage. Reference evidence is most valuable when external artifacts are transferable into implementation structure, while implementation requirements are most valuable when fidelity depends on non-obvious paper-specific obligations. Small gaps are therefore ambiguous: they may reflect strong public priors, limited reference transferability, or shared bottlenecks across variants, rather than evidence that the channels are unnecessary. Appendix D Prompting and Structured Outputs The production prompts use a shared system-user pattern. The current pipeline uses schema-constrained JSON for planning stages, deterministic evidence grounding for package-level evidence bundles, and repository-edit prompts for generation and repair. We therefore report compact schema excerpts rather than the full prompt text. The released repository contains the full prompt templates and structured-output schemas. ⬇ "unit_id": "unit_002", "type": "method | protocol | claim | artifact | task", "statement": "paper-specific implementation obligation", "hypothesis": "what claim or mechanism this unit tests", "decision_value": "what implementation decision depends on it", "paper_evidence": ["source paper/addendum excerpt"], "source_paragraph_ids": ["chunk_006_01"], "reference_query_terms": ["optional explicit method or repository cue"], "implementation_surfaces": ["model_or_method", "metric_formula"], "code_obligations": ["concrete executable obligation"], "runtime_interfaces": ["CLI/function/config surface"], "expected_artifacts": ["results/metrics.json"], "suggested_module_kinds": ["training_loop", "artifact_writer"], "implementation_notes": ["inventory or scope notes"], "tags": ["method", "evaluation"] ⬇ "work_package": "work_package_id": "wp_foa_core", "owned_unit_ids": ["unit_002", "unit_003"], "inventories": "obligation_matrix": ["Method -> file path"], "method_inventory": ["named method or baseline"], "parameter_inventory": ["paper-visible values"] , "scope_boundary": "preserve": ["unit-level obligations that must survive"], "implementation_focus": ["active routes and artifacts"] , "method_obligations": ["what code must expose"] , "file_plan": "target_file": "methods/foa.py", "work_package_id": "wp_foa_core", "owned_units": ["unit_002", "unit_003"], "reference_ids": ["ref or paper chunk ids"], "implementation_surfaces": ["model_or_method"], "defines_symbols": ["FOA", "CMAES"], "calls_symbols": ["fitness_function"], "writes_artifacts": ["results/adaptation_trace.json"], "scope_boundary": "preserve": ["file-owned obligations"], "generation_prompt": "file-local implementation instruction", "review_points": ["semantic and route-closure checks"] ⬇ "evidence_bundle": "work_package_id": "wp_foa_core", "focus": "implementation duty", "owned_unit_ids": ["unit_002", "unit_003"], "evidence_links": [ "unit_id": "unit_002", "ref_id": "ref_001", "file_path": "repo/file.py", "snippet_preview": "…", "why_relevant": "…", "confidence": 0.0 ], "context_summary": ["paper or reference evidence summary"], "grounding_status": "grounded | self_contained | weak | ungrounded" , "repair_ticket": "failure_type": "semantic | runtime | integration", "required_fix_targets": ["methods/foa.py"], "allowed_changes": ["targeted repair scope"], "forbidden_changes": ["requirements already preserved"], "next_fix_scope": ["files to patch"] Appendix E Additional Algorithmic Details E.1 Algorithmic Summary Algorithm 1 ReproAgent pipeline 0: Paper P, request r, references ℛR, rounds T 0: Repository C 1: ←ChunkPaper()S (P) 2: ←SurveyRepos(ℛ)Q (R) 3: ←ExtractRequirements()U (S) 4: ←Deduplicate()U (U) 5: ←PlanWorkPackages()W (U) 6: CheckPreserved(,)CheckPreserved(U,W) 7: ℰ←CollectContentEvidence(,)E (W,Q) 8: ←CollectStructureEvidence()Z (Q) 9: ←BuildContract(,,ℰ,)G (U,W,E,Z) 10: ℱ←PlanFiles()F (G) 11: for f∈TopologicalOrder(ℱ)f (F) do 12: f^←GenerateFile(f,,ℰ) f (f,G,E) 13: end for 14: for t=1t=1 to T do 15: vq←RequirementReview(^,)v_q ( C,U) 16: vr←RuntimeValidation(^)v_r ( C) 17: if vq.passed∧vr.passedv_q.passed v_r.passed then 18: break 19: end if 20: τ←RepairTicket(vq,vr,ℰ)τ (v_q,v_r,E) 21: ^←ApplyRepair(^,τ) C ( C,τ) 22: end for 23: return C E.2 Repair Loop Algorithm 2 Contract-preserving repair loop 0: Repository C, requirements U, contracts G, evidence ℰE, budget T 0: Patched repository C 1: for t=1t=1 to T do 2: s←ReviewRequirements(^,,)s ( C,U,G) 3: r←RunValidation(^)r ( C) 4: if s.passed∧r.passeds.passed r.passed then 5: break 6: end if 7: τ←BuildRepairTicket(s,r,,ℰ)τ (s,r,G,E) 8: τban←PreservedReqs(s) _ban (s) 9: ^←PatchAffectedFiles(^,τ) C ( C,τ) 10: end for 11: return C Appendix F Per-Paper Scores and Generated Repository Sizes This section gives the complete per-paper scores and generated repository sizes behind the full and ablated runs reported in the main results. Paper Claude Gemini w/o ref. w/o req. Score Files Score Files Score Files Score Files Adaptive Pruning 67.8 46 41.9 92 6.2 23 14.3 24 All in One 76.9 77 51.6 48 23.7 16 28.0 36 BAM 61.8 37 48.5 47 20.7 18 30.8 38 BBOX 86.4 280 15.6 70 14.6 15 14.2 75 Bridging Data Gaps 66.0 151 50.6 23 13.9 17 38.4 19 FRE 76.8 23 26.9 32 12.7 14 25.1 20 FTRL 63.9 28 40.1 71 11.7 23 13.4 51 LBCS 65.8 40 42.5 51 34.6 24 23.2 22 LCA on the Line 64.9 30 41.8 22 33.8 18 16.1 15 Mech. Understanding 75.7 201 48.1 26 25.9 14 4.6 20 PINN 81.5 163 61.3 35 52.2 14 53.3 18 RICE 75.1 34 31.0 81 19.9 16 27.8 25 Robust CLIP 63.0 51 29.5 19 15.7 14 25.6 20 Sample-specific Masks 82.0 80 53.8 50 28.0 17 21.3 33 SAPG 86.5 141 29.0 25 21.5 16 24.2 24 Sequential Neural Score Estimation 78.0 30 45.7 14 21.5 13 42.6 26 Stay on Topic with Classifier-Free Guidance 58.5 31 25.3 21 21.2 20 22.2 27 Stochastic Interpolants 80.8 73 39.1 25 21.5 19 29.9 24 Test-Time Model Adaptation 75.6 49 54.9 20 23.8 12 47.1 26 What Will My Model Forget 87.6 23 16.8 22 8.4 12 10.7 19 Mean 73.7 79.4 39.7 39.7 21.6 16.8 25.6 28.1 Table 6: Per-paper scores and generated repository sizes for our full and ablated runs. File counts come from the corresponding run metadata and are reported as process metadata rather than scoring inputs. Appendix G Token, Time, and Cost Diagnostics Tables 7 and 8 report usage metadata for interpreting the reproduction process. Scoring-token counts come from PaperBench judge summaries; when a judge summary stores token usage at leaf level, we sum the recoverable per-leaf metadata from the same graded task tree. Pipeline-token counts, wall-clock time, and agent-call counts use exported run logger summaries when available. Runtime metadata is exported at different granularity across settings; the Claude pipeline-token cell is estimated from matched run logs and is reported only as process metadata, while hours and calls use the recoverable run metadata directly. For the Gemini family, the full run uses more pipeline tokens than either ablation, which is expected because the full setting carries both implementation requirements and reference evidence through planning and repair. Wall-clock time varies with target-paper difficulty and available runtime metadata. Setting Papers Score Judge tok. Pipe tok. Hours Calls ReproAgent / Claude 20 73.7 12,868,729 17,833,795 9.3 35.4 ReproAgent / Gemini full 20 39.7 6,243,588 17,459,007 5.6 11.7 Gemini w/o reference evidence 20 21.6 6,582,787 4,034,007 1.5 13.8 Gemini w/o implementation requirements 20 25.6 4,668,761 4,429,321 1.3 12.6 Table 7: Usage metadata for our runs. Judge-token cells use root-level or summed leaf metadata depending on how the PaperBench summary stores usage; pipeline-token cells use exported run metadata or the estimate described above. Paper Claude judge Gemini judge Gemini pipe No-ref judge No-ref pipe No-req judge No-req pipe Adaptive pruning 5,423,412 2,868,933 23,047,479 1,980,736 2,236,269 3,341,264 5,919,631 All-in-one 5,538,759 3,780,967 9,264,024 3,429,304 3,936,893 2,250,662 2,630,553 BAM 30,766,863 7,768,083 16,097,296 8,827,504 4,969,452 1,892,690 15,027,467 BBOX 8,567,577 774,669 58,268,936 4,903,571 4,131,573 4,373,037 4,509,759 Bridging data gaps 4,746,750 2,300,086 8,820,069 1,864,915 4,264,829 2,412,905 4,406,997 FRE 37,387,691 13,780,748 8,425,041 10,006,741 2,975,272 12,737,673 3,925,572 FTRL 11,530,535 12,388,835 23,625,213 3,976,499 5,255,585 291,099 4,136,571 LBCS 35,890,914 17,312,480 9,390,925 20,577,036 5,455,156 9,045,550 6,665,934 LCA on the line 25,328,353 613,700 8,697,850 18,609,830 4,585,214 4,880,558 3,644,005 Mech. understanding 2,176,841 1,589,700 7,923,157 1,340,680 3,134,806 1,383,850 4,118,063 PINN 7,360,909 5,691,891 13,620,939 7,053,399 4,120,951 6,536,328 4,348,687 RICE 16,389,279 1,550,040 74,534,103 6,929,485 3,739,967 5,148,614 3,261,044 Robust CLIP 5,588,405 2,147,322 16,285,677 3,131,589 3,445,426 2,689,292 2,283,182 Sample-specific masks 9,405,505 2,230,104 18,789,684 2,932,425 4,813,964 604,261 2,426,577 SAPG 4,553,396 3,507,987 8,202,509 3,339,036 3,793,693 2,585,659 4,160,601 Sequential NSE 3,293,860 2,661,812 6,486,552 2,075,601 4,250,141 2,723,479 5,711,514 Stay on topic 3,206,265 1,684,783 13,880,313 2,100,836 4,674,944 159,171 2,992,827 Stochastic interpolants 3,641,790 2,171,919 8,435,217 1,364,707 4,452,320 1,731,008 1,227,333 Test-time adaptation 7,102,606 3,326,170 7,495,412 3,511,711 3,539,717 336,977 5,935,940 What will my model forget 29,474,870 36,721,521 7,889,747 23,700,128 2,903,969 28,251,151 1,254,158 Table 8: Per-paper token metadata. “Judge” is PaperBench grading-token usage, with leaf metadata summed for summaries that store usage at leaf level; “pipe” is generation-token usage from exported run metadata when available. Appendix H Full Pipeline Stage Sequence ReproAgent comprises 15 sub-stages organized into four phases. Table 9 lists every stage with its phase assignment and a brief description. Phase Stage Engineering role Prepare Input normalization Normalize the target, task type, entities, and explicit constraints. Prepare Paper chunking Segment paper text into section-level chunks with overflow handling. Prepare Requirement extraction Extract implementation requirements that need concrete code, artifacts, interfaces, or validation targets. Prepare Reference acquisition Collect reference repositories and survey file trees, README files, and likely reusable modules. Plan Work-package planning Group prepared requirements into implementation-oriented packages such as data, method, training, evaluation, and artifact writing. Plan Package reference evidence Attach package-local content and structure evidence; this supports package decisions but does not decide the final file tree. Plan Reference selection Select actionable reference repositories and evidence items for the current package needs. Plan Pipeline plan Specify package dependencies, runnable routes, and ordering constraints. Plan Global contract synthesis Freeze requirement ownership, evidence bindings, result targets, artifact paths, interfaces, validation checks, and cross-package dependencies. Plan Architecture planning Decide file tree, stack, entrypoints, config surfaces, dependency rules, and package-to-file layout. Plan Package-file planning Refine architecture into per-file contracts with owned requirements, evidence, interfaces, artifacts, and validation hooks. Generate Canonical representation synthesis Consolidate requirements, packages, contracts, architecture, file plans, evidence links, and validator expectations. Generate Local file generation Generate source code file by file following file contracts and dependency ordering. Repair Repair validation Validate through requirement review, contract checks, preflight checks, and sandboxed runtime execution. Repair Repair regeneration Build a targeted repair plan and patch affected files while preserving requirements that already pass. Table 9: Complete stage sequence of the ReproAgent pipeline. Stages are executed in order; within-phase stages may share intermediate artifacts. The Repair phase operates as an iterative loop with a fixed budget within each backbone setting. In each iteration, the system executes validation, diagnosis, and regeneration in sequence. The loop terminates when all validation checks pass or the budget is exhausted. H.1 Process Trace and Readiness Each run records intermediate artifacts for auditing each phase, not only the final repository and score. The stable audit records include prepared units, reference-repository records, work-package plans, contract assertions, file-planning artifacts, validation reports, stage status, quality status, and usage summaries. Table 10 summarizes coarse audit statistics from the 20 Gemini full runs. Stage Audit statistic Mean Median Range Prepare Extracted implementation units per paper 33.8 34.0 28–38 Prepare Candidate reference repositories requested per paper 24.6 25.0 10–30 Prepare Prepared reference repositories per paper 1.1 0.0 0–9 Prepare Implementation units grounded by references per paper 6.2 0.0 0–29 Plan Work packages per paper 19.6 21.5 3–38 Plan Contract assertions per paper 6.9 7.0 5–10 Generate Generated files per paper 35.5 26.5 14–80 Runtime metadata Pipeline agent calls per paper 11.7 6.0 3–29 Table 10: Coarse audit statistics from the 20 Gemini full runs. These logged counts summarize the scale of preparation, planning, generation, and runtime interaction recorded during execution; they are not inputs to PaperBench judging. These logged counts make the generation process more concrete. Prepare extracts implementation units from the paper itself rather than from the PaperBench rubric, then attempts to ground those units with related repositories when actionable evidence is available. Prepared reference repositories are concentrated in a subset of papers, reflecting the benchmark’s mix of papers with different amounts of reusable prior-work structure. Plan then turns prepared units into work packages and contract assertions, which are contract-shaping objects rather than file-count targets. Generate materializes the file plans into repositories, and Repair audits requirement, trace, artifact, integration, and runtime checks. These logged counts are not inputs to PaperBench judging. All reported Gemini full runs completed repository generation and were submitted to the PaperBench judge. We report them alongside PaperBench scores to make the pipeline trace auditable.