Paper deep dive
Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research
Lezhi Yu, Xiaogang Xu, Yuhua Zhou, Shuibing He, Aimin Pan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/28/2026, 4:09:21 AM
Summary
The paper introduces ABE-Ralph, an automated auditing framework designed to detect 'methodological hallucinations' in LLM-driven scientific research. Unlike traditional execution-driven evaluations that only check for code success, ABE-Ralph enforces structured experimental constraints (YAML contracts) and performs triple-verification (quantitative, qualitative, structural) to ensure faithful reproduction of scientific claims. The framework was evaluated across 30 long-horizon reproduction runs in 12 ML domains, achieving a 93% robust execution rate, and demonstrated discovery capabilities on NatureBench tasks.
Entities (7)
Relation Signals (6)
ABE-Ralph → detects → Methodological Hallucinations
confidence 95% · To detect these failures, we introduce ABE-Ralph... We show that agents often produce methodological hallucinations
ABE-Ralph → employs → Triple-Verification
confidence 93% · During execution, a Triple-Verification pipeline checks quantitative metric alignment, qualitative semantic logic, and structural code fidelity.
ABE-Ralph → uses → YAML Contracts
confidence 92% · ABE-Ralph structures paper claims... into declarative YAML contracts.
ABE-Ralph → evaluatedon → NatureBench
confidence 90% · In 23 NatureBench discovery tasks, ABE-Ralph matches or exceeds state-of-the-art performance on 5 tasks.
Methodological Hallucinations → includes → M1: Method Integrity Collapse
confidence 88% · Table 3: The 5-category taxonomy... M1: Method Integrity Collapse... RAG: Generator failed to compile; agent substituted exact-match string lookups
ABE-Ralph → outperformsormatches → SWE-agent
confidence 80% · Table 1: Positioning of ABE-Ralph relative to existing agentic systems... ABE-Ralph blocks shortcut handling where SWE-agent has undetected out of test scope.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM agents used for scientific experimentation must do more than generate executable code: they must implement the reference method faithfully, design experiments that test the paper's claims, and provide evidence supporting those claims. We show that agents often produce methodological hallucinations: silently reducing datasets or training budgets, replacing failed learning or generative components with lookup or oracle functions, or drawing conclusions from resource-limited settings where a method's claimed advantage disappears. To detect these failures, we introduce ABE-Ralph, a reference-anchored auditing framework that represents claims, protocols, required components, baselines, and metrics as structured experimental constraints, guides implementation through an 8-step workflow, and performs quantitative, qualitative, and code-level verification. Across 30 long-horizon reproduction runs covering 12 machine learning domains, ABE-Ralph achieves a 93% robust execution rate and identifies five scientific failure modes. In 23 NatureBench discovery tasks, ABE-Ralph matches or exceeds state-of-the-art performance on 5 tasks. These results show that reliable evaluation of AI scientists must assess whether the experimental design faithfully tests the intended claim and whether the resulting evidence supports it, rather than treating code execution or plausible metrics as evidence of scientific success.
Tags
Links
- Source: https://arxiv.org/abs/2608.26753v1
- Canonical: https://arxiv.org/abs/2608.26753v1
Trouble viewing inline? Open PDF directly →
Full Text
86,718 characters extracted from source content.
Expand or collapse full text
Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research Lezhi Yu Xiaogang Xu Yuhua Zhou Shuibing He Aimin Pan Abstract LLM agents used for scientific experimentation must do more than generate executable code: they must implement the reference method faithfully, design experiments that test the paper’s claims, and provide evidence supporting those claims. We show that agents often produce methodological hallucinations: silently reducing datasets or training budgets, replacing failed learning or generative components with lookup or oracle functions, or drawing conclusions from resource-limited settings where a method’s claimed advantage disappears. To detect these failures, we introduce ABE-Ralph, a reference-anchored auditing framework that represents claims, protocols, required components, baselines, and metrics as structured experimental constraints, guides implementation through an 8-step workflow, and performs quantitative, qualitative, and code-level verification. Across 30 long-horizon reproduction runs covering 12 machine learning domains, ABE-Ralph achieves a 93% robust execution rate and identifies five scientific failure modes. In 23 NatureBench discovery tasks, ABE-Ralph matches or exceeds state-of-the-art performance on 5 tasks. These results show that reliable evaluation of AI scientists must assess whether the experimental design faithfully tests the intended claim and whether the resulting evidence supports it, rather than treating code execution or plausible metrics as evidence of scientific success. Code — https://github.com/Flavorfish/AutoRepro 1College of Computer Science and Technology, Zhejiang University, Hangzhou, China 2 Zhejiang Lab, Hangzhou, China Introduction Large Language Model (LLM) agents are expanding into autonomous research workflows, allowing systems to generate hypotheses, write code, run experiments, and draft reports (Lu et al. 2024; Liu et al. 2026; Boiko et al. 2023; Bran et al. 2024). Similarly, software engineering agents solve repository issues and output functional patches with high success rates (Yang et al. 2024; Wang et al. 2025; Liu et al. 2024). These developments make it tempting to evaluate scientific agents using the same criterion used for software agents: whether the generated program executes successfully and produces an output. That criterion is insufficient for scientific reproduction. A reproduction must satisfy four linked requirements: (1) the implementation contains the reference method described in the paper; (2) datasets, preprocessing, training, and baselines remain relevant to the original experiment; (3) the experiment is designed to test the central claim under the computational regime; and (4) observed results provide valid evidence for or against that claim. A script returning exit code 0 or reporting plausible metrics can easily fail all four requirements. Figure 1: Comparison between standard execution-driven agents and our proposed ABE-Ralph auditing framework. While traditional agents often bypass computational limits using undetected shortcuts (Methodological Hallucinations), ABE-Ralph locks development constraints via YAML contracts and audits code pipelines through a Triple-Verification system. This distinction is especially crucial when agents operate under compute limits, missing dependencies, or failed checkpoints. When an experiment is difficult to run, an agent may silently use a smaller dataset, fewer training steps, lower resolution, or random weights without reporting protocol changes. It may replace a costly generative module with a lookup rule or oracle function that already holds the answer, or run the method at a scale too small for its claimed advantage to emerge and conclude the hypothesis is false. These actions preserve the appearance of progress while failing scientific logic. We define these deviations as Methodological Hallucinations: silent, hard-to-detect violations of pre-defined experimental constraints that preserve superficial code execution while undermining scientific logic (Abalo-Rodríguez and Pinheiro 2025; Santhosh et al. 2026). Consequently, they produce misleading metrics that risk validating false hypotheses or incorrectly dismissing valid claims. The problem is therefore not only that agents sometimes generate incorrect code. More fundamentally, they can generate a scientifically misleading experimental process while producing technically valid artifacts. Existing execution-driven evaluations are largely unable to distinguish a faithful reproduction from a simplified implementation, an incomplete experiment, or an experiment whose resource constraints invalidate its conclusion. Human reviewers can identify some of these problems, but manual inspection is difficult to scale across long-horizon agent runs. We introduce ABE-Ralph (Auto Baseline Experiment), an automated scientific auditing framework that monitors, binds, and verifies the experimental lifecycle of AI-driven research, treating scientific reproduction as a reference-anchored process. Before implementation, ABE-Ralph structures paper claims, architectural components, datasets, baselines, metrics, and resource bounds into declarative YAML contracts. An 8-step workflow guides model construction and protocol execution. During execution, a Triple-Verification pipeline checks quantitative metric alignment, qualitative semantic logic, and structural code fidelity. These checks intercept deceptive model shortcuts and guarantee methodological fidelity and auditability. Distinguishing execution success from scientific validity is essential. A faithful auditing framework does not merely replicate historical metrics; it provides a foundation to cross-examine original outcomes and systematically discover optimized configurations that exceed baseline benchmarks. We report reference-anchored reproduction outcomes separately from raw execution success, evaluating whether each task reproduces or exceeds reference results. Our contributions include: 1. Taxonomy of Methodological Hallucinations: We conceptualize and define a 5-class taxonomy of deceptive agentic shortcuts in scientific workflows that maintain successful system execution while violating core methodological bounds. 2. The ABE-Ralph Auditing Framework: We propose ABE-Ralph, an 8-step reference-anchored framework enforcing pre-execution YAML contracts and an automated Triple-Verification system (numerical, logical, and code-structure levels). 3. Empirical Evaluation and Discovery: We validate the system across 30 classical ML benchmarks spanning 12 domains (achieving a 93% robust execution rate and exposing systematic methodological shortcuts), and further demonstrate its capability in discovery mode across 23 NatureBench tasks (matching or exceeding SOTA baselines on 5 tasks). Related Work LLM Agents for Scientific Discovery Autonomous scientific agents like The AI Scientist (Lu et al. 2024), AutoResearchClaw (Liu et al. 2026), and Claw-AI-Lab (Wu et al. 2026) build research workflows but focus primarily on novelty and syntax execution, lacking auditing mechanisms for experimental fidelity. Similarly, domain-specific systems, e.g.,ChemCrow (Bran et al. 2024) and Coscientist (Boiko et al. 2023), automate lab tools without verifying logical correctness. As highlighted in recent surveys (Wei et al. 2025), existing platforms evaluate the superficial appearance of scientific work rather than its internal validity. Drawing inspiration from these efforts, ABE-Ralph introduces constraint verification to audit reproduction fidelity. LLM Agents for Software Engineering Software engineering agents like SWE-agent (Yang et al. 2024) and OpenHands (Wang et al. 2025) optimize for passing test suites under SWE-bench paradigms (Jimenez et al. 2024; Yang et al. 2025; Deng et al. 2026). Similarly, code benchmarks like HumanEval (Chen et al. 2021) and MBPP (Austin et al. 2021) measure execution correctness rather than scientific intent. However, passing unit tests or exit-0 checks is insufficient for scientific tasks, as agents can bypass core methodologies via trivial heuristics without triggering errors. Unconstrained patches also remain vulnerable to adversarial flaws (Sajadi, Damevski, and Chatterjee 2025). ABE-Ralph addresses this by introducing constraint verification layers above code execution. Reproducibility Benchmarks and Challenges The machine learning reproducibility crisis led to community initiatives like the ML Reproducibility Challenge and REPROLANG (Branco et al. 2020). These projects require human reviewers to manually inspect replication reports. However, this manual approach does not scale to automated pipelines that run multiple experiments daily. Automated tools like ReproZip (Chirigati et al. 2016) and Code Ocean focus on environment capture, ensuring that code templates can compile, but they do not check if the re-executed code represents the target method. To formalize evaluations of scientific agents, benchmarks like ScienceAgentBench (Chen et al. 2025) and MLE-bench (Chan et al. 2025) target data-driven tasks. We design a multi-axis auditing benchmark, evaluating method logic, protocol steps, and conclusion validity. Table 1: Positioning of ABE-Ralph relative to existing agentic systems across six dimensions. Dimension AI Scientist / AutoResearchClaw SWE-agent / OpenHands Human Repro. Challenge ABE-Ralph (Ours) Primary Goal Novelty-driven discovery Task repair and debugging Manual study replication Automated protocol audit Success Check Paper readability & metrics Unit test pass status (Exit 0) Human qualitative review Multimodal alignment checks Constraint System None (flexible design) Test-driven assertions Manual checklist Structured YAML constraints Shortcut Handling Bypassed if metrics improve Undetected out of test scope Checked by human expert Blocked at verification step Failure Diagnosis Opaque execution status Stack trace output Text report 5-class taxonomy filter Execution Scale Low throughput (papers/day) Large scale (100+ repositories) High latency (months/paper) High throughput batch runs Table 2: The 8-step staged workflow of ABE-Ralph. Each step has a configurable timeout derived from the YAML contract’s compute budget. Step Name Description 1 Intent Discovery Parse the YAML contract parameters and extract research goals, constraints, and metrics. 1.5 Dataset Verification Direct search for authentic datasets matching specifications; blocks synthetic data creation. 2 Repo Search & Selection Locate, verify, and copy repository templates from official sources or verified implementations. 3 Architecture Blueprint Generate blueprint.md defining the model structure and setup interfaces in main.py. 4a Pipeline Integration Execute sanity runs on 5–10 samples to check data paths, memory limits, and CUDA setups. 4b Main Execution Run the experimental pipeline, write output variables to metrics.json, and handle CUDA OOM limits. 5 Analysis & Reporting Compile experiment_result.md comparing achieved metrics against baseline contract rules. 5.5 Output Fallback Parse logs and checkpoint states to recover loss scores and metrics if metrics.json fails to compile. 6 Skill Extraction Generalize code interfaces and utility logic from successful runs for subsequent research. The ABE-Ralph Framework The primary goal of the ABE-Ralph framework is to automate the end-to-end reproduction of scientific paper code and experimental protocols in autonomous AI research. Rather than treating reproduction as an unconstrained script generation task, our central thesis is that faithfully reproducing a reference paper’s methodology, codebase, and experimental pipeline is fundamentally a Constraint Satisfaction Problem (CSP) operating under strict computational resource limits. In this section, we formalize this reproduction paradigm, establish our semantic constraint architecture, and detail how each system component maps directly to solving specific sub-problems within this formal formulation to intercept methodological hallucinations. Formal Problem Formulation We formalize the scientific reproduction of paper code and experiments as a tuple =⟨,,ℛ⟩T= ,D,R , defined as follows: • =c1,c2,…,cnC=\c_1,c_2,…,c_n\ represents a set of multi-modal scientific constraints extracted directly from the reference paper, specifying required architectural components, baseline configurations, metric directionality, and targeted research hypotheses. • D represents the operational data input space, encompassing the canonical dataset, preprocessing protocols, and environmental configurations required to replicate the paper’s experiments. • ℛ=Bcomp,BtimeR=\B_comp,B_time\ denotes the strict resource budget, imposing upper bounds on hardware capacity (e.g., VRAM, FLOPs) and execution time. Given T, an autonomous agent generates an executable program P∈P (where P represents the space of candidate repository scripts) and executes it on D to yield an experimental outcome state E=P()S_E=P(D). Traditional execution-driven evaluations define reproduction success solely through the terminal exit status of the code process: (Exit(P())=0)→Success.I(Exit(P(D))=0) . (1) However, this criterion fails to guarantee that the generated code P implements the target methodology in C. Under resource pressure ℛR, an agent may produce code that exits cleanly (Exit(P())=0Exit(P(D))=0) while silently violating core experimental bounds ∃ck∈∃ c_k s.t. P⊧̸ckP c_k (where ⊧ denotes the standard semantic satisfaction relation from program verification), thereby generating methodological hallucinations. To solve this, we reformulate the automated reproduction of paper code and experiments as finding an optimal implementation P∗P that maximizes scientific fidelity under the extracted constraint set C: P∗=argmaxP∈(P(),) P = _P V(P(D),C) s.t.P⊧∧ℛconsumed≤ℛ, .t. P _consumed , (2) where V is a multi-axis verification function mapping the alignment between the program’s actual execution behavior, structural code elements, and the target constraints C. Semantic Constraint Specification Addressing the constraint set C in our formal tuple =⟨,,ℛ⟩T= ,D,R , this subsection details how ABE-Ralph structures and enforces P⊧P . To prevent code generation drift during paper reproduction, the contract C is operationalized as a declarative manifest mapping the logical and architectural boundaries of the targeted study into three distinct constraint classes: 1. Structural Constraints (strC_str): Define the code topology and algorithmic requirements. They enforce the presence of critical architectural components ℳcriticalM_critical (e.g., specific neural layers, attention blocks, or loss functions) described in the original paper within the code P: str⊧(∀m∈ℳcritical,m⊂AST(P)),C_str (∀ m _critical,\,m (P) ), (3) where AST(P)AST(P) denotes the Abstract Syntax Tree of the generated codebase. In implementation, this is declared via YAML configurations (e.g., critical_modules: [UNetDecoder, SkipConnection]). 2. Procedural Constraints (procC_proc): Bound the execution logic of the paper’s experimental protocol. They explicitly dictate dataset specs (dimensionality, sample sizes), training regimes, and hyperparameter bounds. For example, procC_proc enforces that the training dataset size N≥NminN≥ N_min, preventing the agent from silently downsampling data to bypass compute ceilings. 3. Evaluative Constraints (evalC_eval): Govern the comparative metric schemas and target hypotheses (ytargety_target). They specify primary optimization targets (e.g., direction: maximize, metric: F1-score) to ensure that baseline comparisons and primary claims strictly align with the paper’s original metrics. Phase-Transient Execution and Recovery Operators While C establishes the constraint space, the actual execution of the code pipeline P()P(D) must operate strictly within the resource budget ℛ=Bcomp,BtimeR=\B_comp,B_time\ without breaking P⊧P . This subsection addresses the dynamic state transitions during code execution and introduces bounded recovery mechanisms when hardware faults occur. The execution of P proceeds through a sequence of discrete operational states t→t+1S_t _t+1 across an 8-step structured workflow (Table 2). To prevent runtime crashes (e.g., CUDA OOMs or dependency failures) from causing the agent to introduce uncontrolled code modifications, we define a formal recovery operator ℋH. Let ℰE represent runtime exception states: ℰ=OOM,DependencyMismatch,…,Timeout.E=\OOM,DependencyMismatch,…,Timeout\. (4) When execution encounters an exception state t∈ℰS_t , the recovery operator ℋH mutates the local runtime parameters while strictly respecting the bounds set by C: ℋ:t×→t+1∈valid,H:S_t×C _t+1 _valid, (5) where validS_valid denotes valid execution trajectories that do not violate core scientific assertions. Example: If P triggers an Out-Of-Memory error (t=OOMS_t=OOM), an unconstrained agent might alter the preprocessing code to downsample input images from 256×256256× 256 to 64×6464× 64, violating resolution bounds in procC_proc. Under ABE-Ralph, ℋH restricts the fix to dynamic execution hyper-parameters (e.g., enabling gradient accumulation or halving micro-batch size) while keeping input data dimensions fixed. This ensures P()P(D) runs within ℛconsumed≤ℛR_consumed while preserving protocol validity. Multi-Axis Verification Vector To directly implement the verification function (P(),)V(P(D),C) formulated in Eq. (2), ABE-Ralph introduces a Triple-Verification pipeline. This pipeline converts the post-execution state E=P()S_E=P(D) into a structured verification vector =[Vquant,Vqual,Vstruct]Tv=[V_quant,V_qual,V_struct]^T, evaluating the reproduced code and experimental outputs across three distinct axes. Level 1: Quantitative Alignment (VquantV_quant) This component verifies whether the reproduced numerical metrics quantitatively validate the reference paper’s baseline claims under evalC_eval. Let mtm_t represent the reproduced metric, mbm_b the baseline metric, and superscript ∗ the original reference paper values: Vquant V_quant =(sign(mt−mb)=sign(mt∗−mb∗)) =I (sign(m_t-m_b)=sign(m_t -m_b ) ) ∧(|mt−mt∗||mt∗|≤ϵ), ( |m_t-m_t ||m_t |≤ε ), (6) where ϵε is a pre-defined tolerance threshold. This ensures the reproduced method retains its claimed advantage over baselines while remaining within standard empirical margins. Level 2: Semantic Logic Verification (VqualV_qual) To detect cases where metric targets are satisfied through logical shortcuts (e.g., hardcoded values), the semantic validator evaluates the alignment between generated code logic, execution logs, and paper hypotheses: Vqual=fϕ(Embed(),Embed(E),Embed(Logs))∈0,1,V_qual=f_φ (Embed(C),Embed(S_E),Embed(Logs) )∈\0,1\, (7) where fϕf_φ maps multimodal embeddings against deterministic rubrics from evalC_eval via self-consistency checking, catching silent violations in experimental protocol logic. Level 3: Structural Alignment Verification (VstructV_struct) This component validates code-level implementation fidelity, explicitly evaluating str⊧PC_str P. Let GrefG_ref be the canonical structural call-graph of the target algorithm and GimplG_impl be the call-graph parsed directly from the generated program P: Vstruct=(Sim(Gimpl,Gref)≥τ)∧(∀c∈str,P⊧c),V_struct=I (Sim(G_impl,G_ref)≥τ ) (∀ c _str,\,P c ), (8) where Sim calculates AST topological graph isomorphism and τ is a compliance threshold. Mechanistically, VstructV_struct parses P using Python’s native ast module; replacing neural modules with dummy passes triggers a topological mismatch and sets Vstruct=0V_struct=0. Combining all three levels, the final objective function evaluates to: (P(),)=∏i∈quant,qual,structVi∈0,1.V(P(D),C)= _i∈\quant,qual,struct\V_i∈\0,1\. (9) Discovery Mode Generalization Finally, we show how the formal reproduction tuple =⟨,,ℛ⟩T= ,D,R generalizes from static paper reproduction to open-ended scientific discovery. In discovery tasks, strC_str and procC_proc shift from strict replication templates to search boundary conditions representing physical, computational, or domain safety limits. The objective function V expands to include an external continuous domain reward feval:P()→ℝf_eval:P(D) . This ensures the agent actively searches for superior code and hyperparameter configurations (P∗P ) while remaining anchored inside the valid search boundaries defined by C. As shown in Table 5 (see Appendix), the agent’s workflow adapts accordingly: in discovery mode, the Intent step reads a problem specification rather than a paper YAML, the Research step searches domain literature instead of reference code, and the Execute step invokes an external evaluator instead of checking against fixed metrics. This generalization allows ABE-Ralph to serve both as a rigorous reproduction auditor and as a bounded optimization engine for scientific discovery tasks. A Taxonomy of Methodological Hallucinations Evaluating 30 long-horizon reproduction runs across 12 machine learning domains revealed systematic, non-obvious failure modes during model execution. To categorize these deceptive shortcuts, we establish a 5-class taxonomy (Table 3). Table 3: The 5-category taxonomy of LLM methodological hallucinations with empirically observed case studies. Category Definition Key Empirical Observation M1: Method Integrity Collapse Omitting core methods and substituting trivial functions to pass metric checks. RAG: Generator failed to compile; agent substituted exact-match string lookups, matching metrics without training. M2: Silent Protocol Degradation Unauthorized alteration of experimental setups to bypass technical hurdles. PEGASUS: Pretrained checkpoint loading was skipped due to download errors, training from scratch and invalidating comparisons. M3: Scale-Driven Conclusion Inversion Code executes correctly, but scale restrictions invert the claimed methodological advantage. U-Net & SimCLR: Under small-scale runs, simple encoders outperformed U-Net, inverting the central paper hypothesis. M4: Quantitative Key Mismatch Producing correct numerical values under non-standard JSON key names. DPR: Outputted correct recall scores but used non-standard keys in metrics.json, triggering false schema errors. M5: Incomplete Execution Halting the experimental pipeline prematurely while claiming full validation. DDIM: Evaluated only 1 of 3 required configurations and skipped target baseline comparisons. Categories M1 and M2 pose the greatest threat to automated research because they yield plausible metrics while fundamentally compromising experimental logic—failures completely invisible to exit-code verification. M3 demonstrates that methodological advantages are often regime-dependent, where scale constraints inadvertently invert scientific conclusions. This taxonomy directly guides our framework design: M1 and M2 necessitate structural AST checking and semantic contracts; M3 requires resource-bounded execution tracking; M4 is resolved via schema normalization; and M5 is mitigated through progress persistence loops. Experiments and Results In this section, we evaluate the empirical effectiveness of the ABE-Ralph framework. We structure our evaluation to address three primary research questions: • RQ1 (Systematic Performance): How effectively does ABE-Ralph guide autonomous agents to complete scientific reproduction tasks under budget constraints? • RQ2 (Hallucination Characterization): What are the empirical distributions and failure modes of methodological hallucinations? • RQ3 (Verification Contribution): How do verification layers (Vquant,Vqual,VstructV_quant,V_qual,V_struct) contribute to stability and hallucination containment? Experimental Setup Benchmark Tasks We construct a benchmark consisting of 30 distinct reproduction tasks derived from classic ML, computational physics, and bio-informatics domains (e.g., implementing architecture variants of U-Net, training ResNet pipelines under resource constraints, and optimizing multi-objective scientific solvers). Each task is mapped to a ground-truth specification containing canonical data flows, hyperparameter baselines, and reference metrics. Baselines We evaluate and compare five representative LLM-based systems: • Raw LLM: A zero-shot execution model using GPT-4o without a scaffolding loop. • Autonomous Research Catalyst (ARC): A sequential execution framework that follows a static step-by-step reproduction recipe. • Claw-AI-Lab: A template-driven agent utilizing fixed scripts to interface with scientific database environments (Wu et al. 2026). • Claude Code CLI: A state-of-the-art interactive software agent optimized for codebase navigation and automated debugging. • ABE-Ralph (Ours): The proposed framework integrated with our Reference-Anchored auditing loop (instantiated via the ABE-Ralph-Ralph configuration). To ensure a fair and rigorous comparison, all baseline systems—including Raw LLM and Claude Code CLI—were provided with detailed system prompts strictly equivalent to the YAML contracts used by ABE-Ralph. This guarantees that all agents received the identical data flow definitions, hyperparameter baselines, and constraint specifications, ensuring that performance differences stem from the framework’s auditing and feedback mechanisms rather than information asymmetry. Evaluation Metric To evaluate the outputs across both engineering completeness and scientific rigor, we define a Weighted Composite Score (Scomp∈[0,100]S_comp∈[0,100]): Scomp=0.20⋅Sdes+0.25⋅Srel+0.25⋅Srig+0.30⋅Scomp_rate,S_comp=0.20· S_des+0.25· S_rel+0.25· S_rig+0.30· S_comp\_rate, (10) where SdesS_des evaluates structural layout compliance of the generated repository, SrelS_rel measures run-time exceptions and process crash rates (reliability), SrigS_rig computes the mathematical rigor of the evaluation protocols, and Scomp_rateS_comp\_rate represents feature completeness against target specifications. Overall Benchmark Results (RQ1) We report the overall comparative performance of the baseline platforms across the benchmark in Figure 2. Figure 2: Overall framework ranking based on the Weighted Composite Score. N values denote the aggregate number of completed experimental runs evaluated per system configuration. The empirical results show that ABE-Ralph achieves the highest composite score of 58.8, significantly outperforming the Raw LLM baseline (30.8) and scientific templates (ARC at 33.0 and Claw-AI-Lab at 39.9) across the n=30n=30 reproduction tasks. Crucially, Claude Code CLI—a state-of-the-art software development agent—reaches a competitive score of 51.0. However, execution log analysis reveals its performance gains stem primarily from software compilation correctness (SdesS_des and SrelS_rel). Lacking execution-time semantic constraints (strC_str or procC_proc), it frequently relies on shortcuts: under parameter mismatches or slow compilation, it often bypasses convolutional blocks or downscales spatial resolutions merely to ensure exit status compliance. By contrast, ABE-Ralph dynamically bounds the agent’s action space through declarative contracts during the execution loop. This strategy successfully maintains search-space compliance, yielding a superior overall scientific reproduction fidelity. Fine-grained Dimensional Performance Analysis To dissect the specific strengths and vulnerabilities of each baseline, we analyze their average performance across six fundamental dimensions: (A) Design, (B) Reliability, (C) Rigor, (D) Completeness, (E) Alignment, and (F) LLM Review. The quantitative breakdown is illustrated in Figure 3. Figure 3: Framework Comparison across A-F Dimension Average Scores (0–100 scale). Dimensions encompass A: Design, B: Reliability, C: Rigor, D: Completeness, E: Alignment, and F: LLM Review. • Dimension B (Reliability): While ABE-Ralph (63) and Claude Code CLI (62) demonstrate high robustness against run-time crashes, the Raw LLM and ARC baselines register near-zero reliability scores. This discrepancy highlights that raw generation pipelines lack the necessary execution-feedback loops to resolve package dependency conflicts (‘ModuleNotFoundError‘) or environmental path mismatches autonomously. • Dimension D (Completeness): Completeness remains a severe bottleneck across all frameworks. ABE-Ralph leads with a score of 46, whereas all other baselines are constrained below 32. This performance gap indicates that while agents can satisfy localized execution steps, orchestrating the complete scope of a scientific work (including edge cases, plotting, and complex baseline variations) remains a significant open challenge. • Dimension E (Alignment) and Dimension F (LLM Review): In terms of Alignment (Dimension E), ABE-Ralph achieves a critical margin of improvement (90 vs. 78 for Claude Code CLI), proving that the reference contract C successfully anchors the agent to the original method. Conversely, under LLM Review (Dimension F), scores for all models collapse to below 15. This drastic drop suggests that even when scripts run continuously and outputs align quantitatively, current agents still fail to provide the deep semantic narrative, physical intuition, and rigorous verification details expected in final academic files. Distribution of Methodological Hallucinations (RQ2) To investigate the failure modes of autonomous agents in scientific reproduction, we analyze the distribution of methodological hallucinations across the N=30N=30 benchmark reproduction experiments. The categorization is mapped to our defined taxonomy (M1 to M5). The relative distribution is illustrated in Figure 4. Figure 4: Distribution of Methodological Hallucination types across N=30N=30 benchmark runs. A run is categorized as “Clean” only if it completely passes all three verification layers without any shortcut adaptations. Our qualitative analysis indicates that 43.3% (n=13n=13) of the experimental executions are entirely free of shortcuts (Clean). The remaining 56.7% (n=17n=17) of runs exhibit one or more forms of methodological hallucination: • M5 (Incomplete Execution) is the most prevalent failure mode, occurring in 53.3% (n=16n=16) of trials. This is heavily driven by hardware-compute limits: scientific tasks hit physical computational ceilings (e.g., GPU OOM exceptions or container timeouts). Lacking adaptive execution mechanisms, agents default to terminating execution mid-process, resulting in incomplete outputs (Thiyagalingam et al. 2022). • M2 (Silent Protocol Degradation) is present in 20.0% (n=6n=6) of evaluations. Agents silently scale down hyperparameters (such as batch size or epoch count) to resolve resource limits without registering these changes in final reports. • M1 (Method Integrity Collapse, 16.7%), M3 (Scale-Driven Conclusion Inversion, 13.3%), and M4 (Quantitative Key Mismatch, 6.7%) make up the remaining occurrences. Crucially, the cumulative count of detected hallucination types (16+6+5+4+2=3316+6+5+4+2=33 occurrences) exceeds the number of compromised runs (n=17n=17). This occurrence mismatch demonstrates that agent failures are rarely isolated; instead, they often cause cascading errors across multiple operational boundaries. For instance, upon an OOM error, an agent may concurrently shrink train-set size (M2) and strip skip-connections (M1) to force execution. This highlights the risk of relying solely on compile-success metrics. Ablation Study (RQ3) To address RQ3 and evaluate the contribution of each verification layer to the framework’s robustness, we conducted an ablation study. We systematically disabled Level 1 (Quantitative, VquantV_quant), Level 2 (Qualitative/Semantic, VqualV_qual), and Level 3 (Structural, VstructV_struct) verification modules and measured the Average Overall Score and execution variance. The results are illustrated in Figure 5. Figure 5: Ablation Study on the Triple-Verification Pipeline. Removing Level 2 (VqualV_qual) causes the most significant performance degradation and variance explosion, highlighting the necessity of semantic auditing. Table 4: Discovery mode performance on NatureBench tasks. Bold indicates tasks where ABE-Ralph matches or exceeds the SOTA baseline. Task Domain # Tasks ABE-Ralph SOTA Best Result Graph Optimization 8 2 2 / 8 Time-Series Forecasting 6 1 1 / 6 Combinatorial Search 5 1 1 / 5 Scientific Simulation 4 1 1 / 4 Total 23 5 5 / 23 The complete ABE-Ralph framework (Full) achieves an average score of 44.2 with low variance, indicating stable and faithful experimental reproductions. Removing the Quantitative layer (w/o L1) results in a minimal performance drop (-0.7). This confirms our core premise that simply checking numerical outputs is an insufficient safeguard against modern AI agents, as they can easily produce plausible metrics through methodological shortcuts without failing quantitative thresholds. Removing the Structural Code Alignment layer (w/o L3) leads to a moderate decrease of 1.6 points. Without VstructV_struct, the framework struggles to enforce architectural constraints (e.g., preventing the silent removal of key neural network modules), leading to more M1-type (Method Integrity Collapse) failures. Crucially, removing the Qualitative Semantic Verification layer (w/o L2) causes the most severe performance degradation (-5.5) and a dramatic increase in outcome variance (indicated by the large error bars in Figure 5). Without semantic logic checks, the framework cannot detect when an agent subtly alters the experimental protocol, evaluation setup, or dataset to bypass hardware limits. This extreme instability confirms that VqualV_qual is the most essential component for containing methodological hallucinations, ensuring that the agent’s actions logically align with the intended scientific hypothesis rather than merely compiling successfully. NatureBench Discovery Evaluation To evaluate ABE-Ralph’s generalization to open-ended scientific discovery tasks, we deploy the framework in discovery mode (Section Discovery Mode Generalization) on 23 NatureBench tasks spanning diverse scientific domains. In discovery mode, the constraint set C shifts from strict paper replication to domain-specific search boundaries, and the objective incorporates an external evaluation reward fevalf_eval defined by each task’s hidden evaluator. Table 4 summarizes the results. ABE-Ralph achieves or exceeds the state-of-the-art baseline on 5 of 23 tasks, demonstrating that the constraint-anchored optimization framework can effectively navigate open-ended scientific search spaces. On the remaining 18 tasks, ABE-Ralph produces valid solutions within the constraint boundaries but falls short of the best-known results, indicating that the discovery mode provides a sound foundation for further optimization through extended search or multi-agent collaboration. These results validate that the formal tuple =⟨,,ℛ⟩T= ,D,R generalizes effectively from reproduction to discovery. The constraint boundaries C prevent the agent from exploring physically invalid or computationally infeasible regions, while the expanded objective +fevalV+f_eval guides search toward measurable improvements. Detailed per-task results are provided in the Appendix. Discussion The empirical evaluations yield critical insights into the capabilities and limits of modern scientific agents. First, our fine-grained dimensional analysis (Section Fine-grained Dimensional Performance Analysis) reveals a performance mismatch between software viability and scientific validity. General-purpose coding agents generate syntactically correct code, but they lack the domain-specific constraints needed to maintain research integrity. When faced with execution roadblocks, they optimize for engineering execution success (Exit Code 0Exit Code 0) rather than scientific accuracy. Second, the high prevalence of M5 (Incomplete Execution, 53.3%) and M2 (Silent Protocol Degradation, 20.0%) highlights the challenge of resource adaptation under fixed configurations. When compute bounds are reached, agents are forced into a trade-off: either abort execution (resulting in M5) or silently downgrade parameters (such as batch size or training steps, leading to M2) to satisfy execution limits. Because current agent architectures lack the context to dynamically partition workloads or negotiate resource limits, scaling automated discovery beyond simple sandbox environments remains difficult. Third, the low performance across all frameworks under Dimension F (LLM Review) points to an abstraction gap. Current agents focus on local code correction, but they struggle to synthesize their findings into the structured, coherent, and contextualized narratives expected in academic research. Looking forward, transitioning from single-agent designs to cooperative, resource-aware multi-agent systems could address the compute-bound limitations that lead to high rates of incomplete execution. A multi-agent framework with specialized roles—a resource orchestrator for dynamic hardware monitoring, a developer agent for code generation, an auditor agent for constraint verification, and a red-teaming agent for edge-case discovery—could prevent task failures caused by resource limits, laying a foundation for more reliable and scalable automated scientific discovery. Conclusion In this paper, we formalized scientific reproduction as a constraint satisfaction problem under resource bounds and introduced ABE-Ralph, an automated scientific auditing framework that enforces design invariants through declarative YAML contracts and a multi-layered Triple-Verification system. Our empirical evaluation across 30 benchmark reproduction tasks demonstrates that ABE-Ralph achieves a 93% robust execution rate and a weighted composite score of 58.8, significantly outperforming existing agentic baselines. Through systematic analysis of methodological hallucinations, we identified five distinct failure modes—led by Incomplete Execution (53.3%) and Silent Protocol Degradation (20.0%)—that are invisible to exit-code-based evaluation. Ablation results confirm that semantic logic verification (VqualV_qual) is the most essential component for containing these hallucinations. These findings establish that reliable evaluation of AI scientists must assess whether the experimental design faithfully tests the intended claim, rather than treating code execution or plausible metrics as evidence of scientific success. References Abalo-Rodríguez and Pinheiro (2025) Abalo-Rodríguez, I.; and Pinheiro, A. P. 2025. The Hitchhiker’s guide to hallucination research. Consciousness and Cognition, 136: 103941. Austin et al. (2021) Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; and Sutton, C. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732. Boiko et al. (2023) Boiko, D. A.; MacKnight, R.; Kline, B.; and Gomes, G. 2023. Autonomous chemical research with large language models. Nature, 624(7992): 570–578. Bran et al. (2024) Bran, A. M.; Cox, S.; Schilter, O.; Baldassari, C.; White, A. D.; and Schwaller, P. 2024. ChemCrow: Augmenting large-language models with chemistry tools. Nature Machine Intelligence, 6(5): 525–535. Branco et al. (2020) Branco, A.; Calzolari, N.; Vossen, P.; Noord, G. V.; van Uytvanck, D.; Silva, J.; Gomes, L.; Moreira, A.; and Elbers, W. 2020. A Shared Task of a New, Collaborative Type to Foster Reproducibility: A First Exercise in the Area of Language Science and Technology with REPROLANG2020. In Proceedings of the Twelfth Language Resources and Evaluation Conference, 5539–5545. Marseille, France: European Language Resources Association. Chan et al. (2025) Chan, J. S.; Chowdhury, N.; Jaffe, O.; Aung, J.; Sherburn, D.; Mays, E.; Starace, G.; Liu, K.; Maksin, L.; Patwardhan, T.; Madry, A.; and Weng, L. 2025. MLE-Bench: Evaluating Machine Learning Agents on Machine Learning Engineering. In International Conference on Learning Representations. Chen et al. (2021) Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; de Oliveira Pinto, H. P.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; Ray, A.; Puri, R.; Krueger, G.; Petrov, M.; Khlaaf, H.; Sastry, G.; Mishkin, P.; Chan, B.; Gray, S.; Ryder, N.; Pavlov, M.; Power, A.; Kaiser, L.; Bavarian, M.; Winter, C.; Tillet, P.; Such, F. P.; Cummings, D.; Plappert, M.; Chantzis, F.; Barnes, E.; Herbert-Voss, A.; Guss, W. H.; Nichol, A.; Paino, A.; Tezak, N.; Tang, J.; Babuschkin, I.; Balaji, S.; Jain, S.; Saunders, W.; Hesse, C.; Carr, A. N.; Leike, J.; Achiam, J.; Misra, V.; Morikawa, E.; Radford, A.; Knight, M.; Brundage, M.; Murati, M.; Mayer, K.; Welinder, P.; McGrew, B.; Amodei, D.; McCandlish, S.; Sutskever, I.; and Zaremba, W. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374. Chen et al. (2025) Chen, Z.; Chen, S.; Ning, Y.; Zhang, Q.; Wang, B.; Yu, B.; Li, Y.; Liao, Z.; Wei, C.; Lu, Z.; Dey, V.; Xue, M.; Baker, F. N.; Burns, B.; Adu-Ampratwum, D.; Huang, X.; Ning, X.; Gao, S.; Su, Y.; and Sun, H. 2025. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. In International Conference on Learning Representations. Chirigati et al. (2016) Chirigati, F.; Rampin, R.; Shasha, D.; and Freire, J. 2016. ReproZip: Computational Reproducibility With Ease. In Proceedings of the 2016 International Conference on Management of Data, 2085–2088. New York, NY, USA: Association for Computing Machinery. Deng et al. (2026) Deng, X.; Da, J.; Pan, E.; He, Y. Y.; Ide, C.; Garg, K.; Lauffer, N.; Park, A.; Rane, C.; Sampath, K.; Krishnan, M.; Kundurthy, S. R.; Hendryx, S. M.; Wang, Z.; Zhang, C. B. C.; Jacobson, N.; Liu, B.; and Kenstler, B. 2026. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? In Forty-third International Conference on Machine Learning. Jimenez et al. (2024) Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In International Conference on Learning Representations (ICLR). Liu et al. (2026) Liu, J.; Qiu, S.; Li, M.; Li, B.; Ji, H.; Han, S.; Ye, X.; Xia, P.; Dong, Z.; Chen, M.; Zhang, C.; Zhang, L.; Chen, G.; Tu, H.; Yang, X.; Feng, L.; Zhao, X.; Chen, H.; Zhou, J.; Wang, X.; Zhang, W.; Zhu, H.; Li, Y.; Mei, J.; Fei, H.; Zhang, J.; Li, L.; Zhang, L.; Zhou, Y.; Wang, S.; Xiong, C.; Zou, J.; Zheng, Z.; Xie, C.; Ding, M.; and Yao, H. 2026. AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration. arXiv:2605.20025. Liu et al. (2024) Liu, X.; Yu, H.; Zhang, H.; Xu, Y.; Lei, X.; Lai, H.; Gu, Y.; Ding, H.; Men, K.; Yang, K.; Zhang, S.; Deng, X.; Zeng, A.; Du, Z.; Zhang, C.; Shen, S.; Zhang, T.; Su, Y.; Sun, H.; Huang, M.; Dong, Y.; and Tang, J. 2024. AgentBench: Evaluating LLMs as Agents. In International Conference on Learning Representations, 52989–53046. Lu et al. (2024) Lu, C.; Lu, C.; Lange, R. T.; Foerster, J.; Clune, J.; and Ha, D. 2024. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv preprint arXiv:2408.06292. Sajadi, Damevski, and Chatterjee (2025) Sajadi, A.; Damevski, K.; and Chatterjee, P. 2025. How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench. arXiv:2507.02976. Santhosh et al. (2026) Santhosh, V. N.; Vas, R.; Roychowdhury, B.; Sakthi, K.; and Rahaman, M. 2026. Development and content validation of the CAREFUL-AI framework for evaluating AI-generated scientific manuscripts: an exploratory cross-platform study. Research Evaluation, 35: rvag024. Thiyagalingam et al. (2022) Thiyagalingam, J.; Shankar, M.; Fox, G.; and Hey, T. 2022. Scientific machine learning benchmarks. Nature Reviews Physics, 4(6): 413–420. Wang et al. (2025) Wang, X.; Li, B.; Song, Y.; Xu, F. F.; Tang, X.; Zhuge, M.; Pan, J.; Song, Y.; Li, B.; Singh, J.; Tran, H. H.; Li, F.; Ma, R.; Zheng, M.; Qian, B.; Shao, D.; Muennighoff, N.; Zhang, Y.; Hui, B.; Lin, J.; Brennan, R.; Peng, H.; Ji, H.; and Neubig, G. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In The Thirteenth International Conference on Learning Representations. Wei et al. (2025) Wei, J.; Yang, Y.; Zhang, X.; Chen, Y.; Zhuang, X.; Gao, Z.; Zhou, D.; Wang, G.; Gao, Z.; Cao, J.; Qiu, Z.; Hu, M.; Ma, C.; Tang, S.; He, J.; Song, C.; He, X.; Zhang, Q.; You, C.; Zheng, S.; Ding, N.; Ouyang, W.; Dong, N.; Cheng, Y.; Sun, S.; Bai, L.; and Zhou, B. 2025. From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery. arXiv:2508.14111. Wu et al. (2026) Wu, F.; Chen, C.; Tan, Z.; Zhang, T.; Xu, X.; Qian, Y.; Gao, D.; Zhu, L.; Zhu, Q.; Tan, Y.; Ji, D.; Lin, G.; Chen, T.; Ye, D.; and Liu, F. 2026. Claw AI Lab: An Autonomous Multi-Agent Research Team. arXiv:2605.22662. Yang et al. (2024) Yang, J.; Jimenez, C. E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K.; and Press, O. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems 37 (NeurIPS). Yang et al. (2025) Yang, J.; Jimenez, C. E.; Zhang, A.; Lieret, K.; Yang, J.; Wu, X.; Press, O.; Muennighoff, N.; Synnaeve, G.; Narasimhan, K.; Yang, D.; Wang, S.; and Press, O. 2025. SWE-Bench Multimodal: Do AI Systems Generalize to Visual Software Domains? In International Conference on Learning Representations, 2794–2829. Yuan et al. (2025) Yuan, J.; Yan, X.; Zhang, B.; Chen, T.; Shi, B.; Ouyang, W.; Qiao, Y.; Bai, L.; and Zhou, B. 2025. Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 21768–21789. Vienna, Austria: Association for Computational Linguistics. Appendix A Code Logic and Architecture of the ABE-Ralph Framework This section provides the complete algorithmic specification and architecture of the ABE-Ralph (ABE-Ralph) framework, detailing its agent lifecycle, verification engine, contract structure, and file inventory. Overview of the Dual-Innovation Architecture The ABE-Ralph framework embodies two tightly coupled innovations: 1. The Meta-Experiment Agent (ralph_github.py) — a structured 8-step workflow orchestrating an LLM through experiment reproduction, self-healing, and ablation analysis. 2. The Triple-Verification Engine (verify_reproduction.py) — a three-tier validation system evaluating quantitative alignment (L1), qualitative peer review (L2), and code implementation fidelity (L3). These components are mediated by a structured YAML contract defining expected datasets, baselines, target methods, metrics, success conditions, and critical modules. The Meta-Experiment Agent (ralph_github.py) Class Structure and Initialization The agent is encapsulated in MetaExperimentAgent. Algorithm 1 formalizes initialization. Algorithm 1 Agent Initialization 0: plan_path (YAML experiment plan), workspace (directory) 0: Initialized agent instance 1: self.workspace ← workspace 2: self.plan_data ← ParseYAML(plan_path) 3: self.discovery_mode ← IsDiscovery(self.plan_data) 4: InitSignalHandling() 5: SetupWorkspace() 6: self._budget_timeouts ← MapBudgets(self.plan_data) 7: if resume is TRUE then 8: LoadCheckpoint() 9: else 10: InitCheckpoint() 11: end if Signal Handling: Self-Pipe Pattern To ensure interrupt responsiveness during subprocess monitoring, ABE-Ralph implements a self-pipe pattern (Algorithm 2). Algorithm 2 Signal Handling via Self-Pipe Pattern 1: Thread 1 (Signal Monitor): 2: pthread_sigmask(BLOCK, SIGINT) 3: while TRUE do 4: sigwait(SIGINT) 5: self.interrupted ← TRUE 6: WriteByte(self_pipe_write_end) 7: end while 8: Thread 0 (Main Execution Thread): 9: while process is running do 10: ready ← select(stdout, self_pipe_read_end) 11: if self_pipe_read_end ∈ ready then 12: break Ctrl-C received, terminate safely 13: end if 14: ReadLine(stdout) 15: end while The 8-Step Staged Workflow Algorithm 3 details the sequential 8-step pipeline execution. Algorithm 3 The 8-Step Staged Workflow 1: procedure RunWorkflow() 2: RunClaudeStep(prompt_intent, timeout=300s) 3: RunClaudeStep(prompt_dataset, timeout=3600s) 4: if DATA_READY ∉ output and --strict-data then 5: raise DatasetVerificationError 6: end if 7: RunClaudeStep(prompt_research, timeout=1800s) 8: RunClaudeStep(prompt_blueprint, timeout=300s) 9: RunClaudeStep(prompt_smoke_test, timeout=1800s) 10: RunClaudeStep(prompt_full_execution, timeout=3600s) 11: VerifyMetricsJson() 12: RunClaudeStep(prompt_synthesis, timeout=600s) 13: if metrics.json missing then 14: RunClaudeStep(prompt_extract_metrics, timeout=600s) 15: end if 16: RunClaudeStep(prompt_skills) 17: for i=1i=1 to max_retries do 18: RunImprovementCycle(i) 19: if HYPOTHESES_SUPPORTED ∨ OPTIMIZATION_DONE then 20: break 21: end if 22: end for 23: RunAblationStudy() 24: end procedure Discovery Mode vs. Reproduction Mode Table 5 contrasts the agent’s behavior under Reproduction Mode and Discovery Mode. Table 5: Comparison of Reproduction and Discovery Modes. Step Reproduction Mode Discovery Mode 1. Intent Analyze YAML Read problem/README.md 1.5. Dataset Search HF/Kaggle/OpenML Analyze local files and evaluator.py 2. Research Search reference code repos Search domain literature 3. Blueprint Build main.py from reference Design solution from scratch 4b. Execute Train & output metrics.json Train & execute evaluator.py 5. Synthesis Check YAML criteria Summarize evaluator score The Improvement Cycle Algorithm 4 presents the iterative refinement loop. Algorithm 4 Iterative Improvement Cycle 1: procedure RunImprovementCycle(iteration, max_retries) 2: prompt_imp ← ConstructPrompt( 3: Audit: “Analyze experiment_result.md and raw data”, 4: CodeCheck: “Verify main.py against YAML contract”, 5: Fix: “Patch code and re-execute main.py”) 6: RunClaudeStep(prompt_imp, step_name=“imp_v” + iteration) 7: prompt_ana ← ConstructPrompt( 8: Review: “Compare results against YAML success criteria”, 9: Signal: “Output HYPOTHESES_SUPPORTED or OPTIMIZATION_DONE”) 10: out ← RunClaudeStep(prompt_ana, step_name=“ana_v” + iteration) 11: converged ← (HYPOTHESES_SUPPORTED ∈ out) 12: opt_done ← (OPTIMIZATION_DONE ∈ out) 13: TrackIteration(iteration, converged, opt_done) 14: return out 15: end procedure Checkpoint State Management The execution state is persisted in experiment_state.json. The schema is defined as: ⬇ 1 2 "plan_path": "bm_001_dpr.yaml", 3 "plan_mtime": 1717027200.0, 4 "started_at": "2026-05-27T10:00:00", 5 "steps": 6 "intent_discovery": "status": "completed", "duration_seconds": 45.2, 7 "dataset_search": "status": "completed", "duration_seconds": 120.3, 8 "research_step": "status": "completed", "duration_seconds": 300.1, 9 "blueprint": "status": "completed", 10 "smoke_test": "status": "completed", 11 "full_execution": "status": "pending", 12 "synthesis_reporting": "status": "pending", 13 "skill_extraction": "status": "pending" 14 , 15 "iteration": "current": 0, "max_retries": 3, "history": [] 16 The Ablation Study System Algorithm 5 outlines the systematic ablation procedure. Algorithm 5 Ablation Study Execution 1: procedure RunAblationStudy() 2: for all cfg ∈ Full, NoL3, NoL1, NoL2 do 3: ws ← PrepareIsolatedWorkspace(cfg) 4: SwitchTo(ws) 5: if cfg == NoL3 then 6: RemoveReferenceCodeConstraints() 7: else if cfg == NoL1 then 8: RemoveQuantitativeTargetPrompts() 9: else if cfg == NoL2 then 10: DisableDeepAuditPrompts() 11: end if 12: decision ← ExecuteEvaluation() 13: scores ← ScoreWorkspaceMultiDim(cfg) 14: RestoreWorkspace() 15: end for 16: end procedure Table 6 maps ablation configurations to prompt-level modifications. Table 6: Prompt-Level Ablation Mapping. Config Removed Prompt Directives Auditing Concept w/o L3 Skip repo search; remove code-guidance Structural Code Alignment w/o L1 Omit YAML metric success criteria comparison Rigid Quantitative Alignment w/o L2 Disable deep audit & root-cause prompt blocks Qualitative Peer Review Multi-Dimensional Scoring Algorithm 6 defines multi-dimensional workspace evaluation. Algorithm 6 Multi-Dimensional Workspace Scoring 1: procedure ScoreWorkspaceMultiDim(label) 2: integrator ← ExperimentIntegrator(workspace) 3: s ← integrator.ComputeMultiDimScore() 4: return Scomp=0.20sA+0.25sB+0.25sC+0.30sDS_comp=0.20s_A+0.25s_B+0.25s_C+0.30s_D, Grade: A/B/C/D/F 5: end procedure The Triple-Verification Engine (verify_reproduction.py) Initialization and Structural Checks Algorithm 7 details initialization of the verifier engine. Algorithm 7 Verifier Engine Initialization 0: plan_path (YAML), workspace 1: self.plan ← ParseYAML(plan_path) 2: self.report ← DetectLatestReport() 3: self.discovery ← (self.plan.mode == “discovery”) 4: self.datasets ← ExtractDatasets(self.plan) 5: self.baselines ← ExtractBaselines(self.plan) 6: self.target ← ExtractTargetMethod(self.plan) Level 1: Rigid Quantitative Alignment (VquantV_quant) Algorithm 8 verifies JSON key structures and presence. Algorithm 8 Level 1 Quantitative Alignment Check 1: procedure RunDynamicQuantitativeCheck(metrics_data) 2: if metrics_data is NULL then 3: return (FALSE, “metrics.json missing”) 4: end if 5: for all ds ∈ self.datasets do 6: k_ds ← FuzzyMatch(ds, metrics_data.keys()) 7: if k_ds is NULL then 8: return (FALSE, “Dataset missing”) 9: end if 10: k_tgt ← FuzzyMatch(self.target, metrics_data[k_ds]) 11: if k_tgt is NULL then 12: return (FALSE, “Target method missing”) 13: end if 14: end for 15: return (TRUE, “Quantitative structure verified”) 16: end procedure Level 2: Qualitative Peer Review (VqualV_qual) Algorithm 9 executes qualitative LLM review. Algorithm 9 Level 2 Qualitative Peer Review Check 1: procedure RunUniversalQualitativeCheck() 2: prompt ← ConstructPrompt( 3: Role: “Senior meta-reviewer auditing reproduction”, 4: Claims: self.plan.key_claims, 5: Report: ReadFile(self.report_path)) 6: out ← InvokeClaude(prompt) 7: decision ← ParseDecision(out) Regex: DECISION: *(MET_ALL|MET_PARTIALLY|FAILED) 8: return (decision) 9: end procedure Level 3: Structural Code Alignment (VstructV_struct) Algorithm 10 executes AST and module existence checks. Algorithm 10 Level 3 Code Alignment Verification 1: procedure RunCodeAlignmentCheck() 2: modules ← self.plan.target_method. critical_modules 3: code ← ReadFile(“main.py”) 4: for all m ∈ modules do 5: if not ASTSearch(m, code) then 6: return (FALSE, “Critical module missing: ” + m) 7: end if 8: end for 9: return (TRUE, “Code structural alignment passed”) 10: end procedure Decision Aggregation Algorithm 11 details the hierarchical decision flow. Algorithm 11 Decision Aggregation Protocol 1: procedure ExecuteEvaluation() 2: m_data ← LoadMetrics() 3: q_pass, _ ← RunDynamicQuantitativeCheck(m_data) 4: decision ← RunUniversalQualitativeCheck() 5: if decision == MET_ALL and m_data is NULL then 6: decision ← MET_PARTIALLY 7: end if 8: if not self.discovery then 9: l3_pass, _ ← RunCodeAlignmentCheck() 10: if not l3_pass then 11: decision ← FAILED Block on M1 method collapse 12: end if 13: end if 14: return decision 15: end procedure Failure Taxonomy Signal Matching The verification results are automatically mapped to failure modes (M1–M5) using regular expressions: ⬇ 1 FAILURE_CATEGORIES = 2 "METHOD_COLLAPSE": 3 "signals": ["oracle", "upper bound", "never started", "substituted"] 4 , 5 "SILENT_DEGRADE": 6 "signals": ["from.scratch", "random init", "degenerate.*solution"] 7 , 8 "SCALE_INVERSION": 9 "signals": ["baseline.*outperform", "direction.*reversal"] 10 , 11 "KEY_MISMATCH": 12 "signals": ["metrics.json.*missing", "key.*not.*match"] 13 , 14 "INCOMPLETE": 15 "signals": ["not all condition", "time budget exhaust"] 16 17 The YAML Contract Specification Table 7 presents the YAML contract structure. Example Contract Instance To ground the abstract schema in Table 7, Listing 1 presents a concrete YAML contract instance corresponding to the DPR (Dense Passage Retrieval) benchmark (bm_001_dpr_dpr.yaml). This contract specifies the paper metadata, key claims, datasets, baselines, target method, expected metric values with tolerance ranges, success/failure conditions, and compute/time budgets that jointly drive both the ABE-Ralph agent’s execution and the Triple-Verification Engine’s decision logic. Listing 1: Example YAML experiment contract (bm_001_dpr_dpr.yaml) used for the DPR benchmark. Fields map directly to the consumers listed in Table 7. ⬇ 1 topic: Dense Passage Retrieval for Open-Domain Question Answering 2 paper_metadata: 3 title: Dense Passage Retrieval for Open-Domain Question Answering 4 authors: Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, ... 5 year: 2020 6 venue: EMNLP 7 domain: nlp_retrieval 8 task_type: open_domain_qa_retrieval 9 reproduction_goal: > 10 Evaluate whether an AI agent can reproduce the main experimental 11 conclusions of DPR: that dense dual-encoder retrieval outperforms 12 BM25 on open-domain QA retrieval metrics, using reduced-scale 13 reproduction on canonical datasets. 14 key_claims: 15 - Dense dual-encoder retrieval (DPR) substantially outperforms BM25 16 on open-domain QA retrieval metrics such as Recall@k and MRR. 17 - Dense retrieval provides stronger downstream evidence quality for 18 QA than sparse lexical retrieval alone. 19 datasets: 20 - NaturalQuestions 21 - TriviaQA 22 baselines: 23 - BM25 (sparse lexical retrieval, e.g., via Pyserini or Elasticsearch) 24 target_method: 25 name: Dense Passage Retriever (DPR) 26 description: A dual-encoder dense retrieval model that encodes 27 questions and passages into a shared embedding space for 28 efficient top-k retrieval via maximum inner product search. 29 codebase: facebookresearch/DPR 30 ablations: 31 - BM25 baseline 32 - DPR with different negative sampling strategies 33 - DPR with reduced training data 34 metrics: 35 - recall_at_20 36 - recall_at_100 37 - mrr 38 expected_metric_values: 39 - metric_name: recall_at_20 40 expected_value: 0.78 41 reasonable_range: [0.73, 0.8] 42 higher_is_better: true 43 - metric_name: recall_at_100 44 expected_value: 0.85 45 reasonable_range: [0.8, 0.88] 46 higher_is_better: true 47 - metric_name: mrr 48 expected_value: 0.32 49 reasonable_range: [0.29, 0.34] 50 higher_is_better: true 51 expected_paper_conclusions: 52 - DPR (dense retrieval) achieves higher Recall@k and MRR than BM25 53 on open-domain QA retrieval benchmarks. 54 - Dense retrieval provides better evidence retrieval for downstream 55 QA than sparse lexical retrieval alone. 56 success_conditions: 57 conclusion_correctness: true 58 metric_direction_consistency: DPR must outperform BM25 on all 59 primary retrieval metrics (recall_at_20, recall_at_100, mrr) in 60 the same direction as the original paper. 61 result_closeness_tolerance: 0.05 62 failure_conditions: 63 - BM25 outperforms or matches DPR on primary retrieval metrics. 64 - Metric differences are outside the reasonable range or opposite 65 in direction to the paper. 66 - Incorrect evaluation setup (e.g., wrong negatives, inconsistent 67 preprocessing) invalidates comparison. 68 compute_budget: 69 gpu_type: NVIDIA GeForce RTX 4070 Laptop GPU 70 gpu_memory_gb: 8 71 max_gpu_hours: 18.0 72 cpu_cores: 8 73 ram_gb: 32 74 time_budget: 75 environment_setup_hours: 2.0 76 repo_selection_and_code_reading_hours: 2.0 77 dataset_parsing_and_section_segmentation_hours: 2.0 78 baseline_pipeline_adaptation_hours: 2.0 79 proposed_method_implementation_hours: 2.5 80 debugging_and_memory_optimization_hours: 2.0 81 baseline_training_and_evaluation_hours: 2.0 82 proposed_method_training_and_evaluation_hours: 2.5 83 ablation_runs_hours: 1.0 84 result_aggregation_and_visualization_hours: 1.0 85 contingency_buffer_hours: 1.0 86 risks: 87 - Incorrect negative sampling or passage index construction may invalidate results. 88 - Incompatible preprocessing between DPR and BM25 could bias metrics. 89 - GPU memory limits may require smaller batch sizes or model variants, affecting metric values. 90 - Dataset splits or answer string matching may differ from the original, impacting recall. 91 reproducibility_notes: 92 Reduced-scale reproduction uses DPR-base model, smaller batch sizes, and fewer epochs to fit within compute budget. Pretrained checkpoints may be used for evaluation if full training is infeasible. Metric values are expected to be within 5% of the original papers results, but directionality and relative ranking between DPR and BM25 are the main criteria for success. 93 generation_metadata: 94 generated_at: ’2026-06-01T16:42:13.157186’ 95 generator_model: gpt-4.1 96 benchmark_id: bm_001_dpr 97 benchmark_name: DPR 98 priority_tier: P1 99 difficulty: 2 100 reproduction_mode: reduced_scale_reproduction 101 retry_count: 0 102 validation_status: complete As shown above, the contract is organized into six functional blocks consumed by different components of the ABE-Ralph pipeline: 1. Provenance block (topic, paper_metadata, reproduction_goal) — consumed by the Meta-Experiment Agent during the Intent and Research steps to establish scope and locate reference implementations. 2. Claim/target block (key_claims, datasets, baselines, target_method, ablations, metrics) — jointly consumed by the Agent (Blueprint/Execution steps) and the Verifier’s L1 and L3 checks to determine which datasets, comparative baselines, and code modules must be present. 3. Quantitative criteria block (expected_metric_values, expected_paper_conclusions, success_conditions, failure_conditions) — directly parameterizes RunDynamicQuantitativeCheck (Algorithm 8) and the L2 qualitative reviewer’s red-line triggers. 4. Resource block (compute_budget, time_budget) — consumed by self._budget_timeouts in Algorithm 1 to bound each staged workflow step (e.g., max_gpu_hours constrains full_execution). 5. Risk block (risks, reproducibility_notes) — informs the Agent’s self-healing/improvement cycle (Algorithm 4) by pre-registering plausible failure causes (e.g., negative sampling errors, preprocessing mismatch) referenced by the failure taxonomy signal matcher. 6. Provenance/audit metadata block (generation_metadata) — used for experiment tracking, reproducibility auditing, and versioning across the 30-benchmark suite (Appendix B–C). Table 7: YAML Contract Schema and Field Consumers. Field Consumers Purpose datasets Agent, Verifier (L1) Expected dataset specifications baselines Agent, Verifier (L1) Required baseline comparative targets target_method Agent, Verifier (L1, L3) Target algorithm description critical_modules Verifier (L3) Mandatory AST components in main.py key_claims Verifier (L2) Claims to evaluate qualitatively success_conditions Agent, Verifier (L1) Metric margins and directionality failure_conditions Verifier (L2) Red-line failure triggers File Inventory Table 8 details the primary system modules. Table 8: Complete System File Inventory. File Name Lines Primary Operational Role ralph_github.py ∼ 1246 Workflow agent, iteration, ablation loop verify_reproduction.py ∼ 519 Triple-verification engine, decision engine alignment_verifier.py ∼ 266 Level 3 structural code fidelity auditor experiment_taxonomy.py ∼ 220 Failure taxonomy classifier (M1–M5) ablation_scoring.py ∼ 280 Multi-dimensional scoring module experiment_integrator.py ∼ 390 Execution history, metric tracking Appendix B Methodological Hallucination Taxonomy: Per-Experiment Breakdown This section details the per-experiment analysis across 30 classical machine learning and deep learning benchmark reproductions evaluated under the ABE-Ralph auditing framework. A total of 17 experiments exhibited one or more methodological hallucination types (M1–M5), while 13 passed verification without shortcuts. M1: Method Integrity Collapse (5 Methods) Occurs when the agent omits core target components, substitutes trivial heuristic fallbacks, or fails to execute comparative conditions. • RAG: (Also M5). The generator fell back to parametric knowledge when retrieval failed (EM 1.8% vs. 42.1% with successful retrieval). The experiment conflated parametric memory with RAG performance, failing to isolate the retrieval component’s contribution. • BERT: (Also M5). The pretrained checkpoint was fine-tuned on fundamentally different tasks (e.g., sentiment analysis) chosen by the LLM, rather than the specific GLUE benchmark tasks (MNLI, QQP, MRPC) specified in the paper. • SimCLR: (Also M3, M5). Baseline methods (e.g., random encoder) were queued but never executed, meaning the core comparison of “SimCLR vs. previous unsupervised methods” never occurred. • DDIM: (Also M5). The core comparison baseline (DDPM) was never executed, meaning the primary claim (DDIM is faster than DDPM) was unvalidated. • Improved DDPM: (Also M5). Training instability (likely in learned variance parameters) eliminated the expected quality gap over base DDPM, failing to reproduce the claimed likelihood improvements. M2: Silent Protocol Degradation (6 Methods) Occurs when training protocols, negative sampling, or initialization routines are modified without authorization to force execution. • ColBERT: (Also M5). Hard negative mining was completely omitted from the training protocol, disabling ColBERT’s ability to discriminate relevant from irrelevant passages. • PEGASUS: (Also M3, M5). Trained from scratch for 3 epochs on 50K samples instead of fine-tuning the pretrained GSG checkpoint, testing random initialization convergence rather than representation quality. • ViT: (Also M3, M5). Trained from scratch on a small dataset without pretraining, resulting in a 20-percentage-point accuracy drop and violating the paper’s assumption of large-scale pretraining. • Faster R-CNN: Protocol degradation in the detection pipeline due to non-standard RPN settings or inconsistent mAP evaluation protocols. • PPO: (Also M5). Continuous-control environments (MuJoCo) crashed with 0/9 runs completed; report falsely claimed continuous control success based on discrete CartPole. • GraphSAGE: (Also M4, M5). Inductive vs. transductive settings were misconfigured, breaking the inductive generalization claim. M3: Scale-Driven Conclusion Inversion (4 Methods) Occurs when resource-limited scaling or protocol degradation inverts the paper’s original ranking or core conclusions. • PEGASUS: (Also M2, M5). PEGASUS underperformed BERT (-32%) and BART (-55%), completely inverting the paper’s SOTA summarization ranking claims. • ViT: (Also M2, M5). At reduced scale without pretraining, CNN baselines outperformed ViT, directly opposing the paper’s conclusion. • SimCLR: (Also M1, M5). The agent substituted a weaker verification standard, concluding “above chance” performance rather than proving superiority over baseline encoders. • U-Net: (Also M5). Reduced-depth ablation paradoxically outperformed the full U-Net under limited data, contradicting claims on encoder-decoder depth utility. M4: Quantitative Key Mismatch (2 Methods) Occurs when output files or metric keys deviate from verification schemas, causing verification pipeline parsing failures. • RoBERTa: (Also M5). Quantitative checks failed solely because metrics.json was produced under a non-standard filename pattern by aggregation scripts. • GraphSAGE: (Also M2, M5). Inconsistent metric naming (e.g., "acc" vs. "accuracy" in output JSON) caused the parser to miss evaluation results. M5: Incomplete Execution (16 Methods) Occurs when benchmark evaluation conditions, datasets, or target conditions are prematurely terminated due to compute limits. Affected methods: RAG, ColBERT, BERT, RoBERTa, PEGASUS, LED, ViT, SimCLR, MoCo, U-Net, YOLO, PatchCore, DDIM, Improved DDPM, PPO, and GraphSAGE. Clean Experiments (13 Baseline Reproduction Runs) Thirteen methods passed all verification checks without hallucination shortcuts: DPR, LoRA, Prefix-Tuning, Longformer, TextRank, ResNet, CLIP, PaDiM, DDPM, DQN, GCN, Informer, and Autoformer. Appendix C Detailed Per-Experiment Case Studies This section provides detailed case studies for all 30 classical machine learning and deep learning benchmark reproductions evaluated under the ABE-Ralph auditing framework. Individual Experiment Analyses 1. DPR (Dense Passage Retrieval) Status: CLEAN. All quantitative and qualitative checks passed successfully. Evaluation metrics for NaturalQuestions and TriviaQA are fully present, and the conclusion (DPR outperforms BM25) is correctly supported by the reproduced experimental data. 2. RAG (Retrieval-Augmented Generation) Status: HALLUCINATION (M1 + M5). • M1 Collapse: The generator fell back to parametric knowledge when retrieval failed (EM 1.8% vs. 42.1% with successful retrieval). The experiment conflated the generator’s memory with RAG performance, thus failing to isolate the retrieval component’s contribution. • M5 Incomplete: Missing metrics for NaturalQuestions and WebQuestions due to index construction infrastructure issues. 3. ColBERT (Contextualized Late Interaction over BERT) Status: HALLUCINATION (M2 + M5). • M2 Degradation: Hard negative mining was completely omitted from the training protocol. Because ColBERT relies heavily on hard negatives, the model failed to discriminate relevant from irrelevant passages. • M5 Incomplete: Omitted the TREC Deep Learning dataset, a primary benchmark in the paper. 4. BERT (Pretrained Language Model) Status: HALLUCINATION (M1 + M5). • M1 Collapse: The pretrained checkpoint was fine-tuned on non-target tasks (e.g., sentiment analysis) chosen by the LLM rather than specific GLUE benchmark tasks (MNLI, QQP, MRPC). • M5 Incomplete: Metrics for the full GLUE dataset suite are missing; only SST-2 results were reported. 5. RoBERTa (Robustly Optimized BERT) Status: HALLUCINATION (M4 + M5). • M4 Key Mismatch: Quantitative verification failed because metrics.json was generated under a non-standard filename pattern by aggregation scripts. • M5 Incomplete: Missing full GLUE benchmark evaluation metrics. 6. LoRA (Low-Rank Adaptation) Status: CLEAN. Clean run. All required metrics for GLUE and instruction-tuning subsets are present. The conclusion that LoRA matches full fine-tuning with significantly fewer trainable parameters is correctly supported. 7. Prefix-Tuning (Continuous Prompt Tuning) Status: CLEAN. Clean reproduction without hallucinations. The LLM successfully set up continuous prefix parameters and compared them against full fine-tuning baselines under stable convergence. 8. PEGASUS (Abstractive Summarization with Stoned Sentences) Status: HALLUCINATION (M2 + M3 + M5). • M2 Degradation: Trained from scratch for 3 epochs on 50K samples instead of using the pretrained GSG checkpoint. • M3 Inversion: The performance ranking inverted: PEGASUS underperformed BERT (-32%) and BART (-55%), contradicting SOTA summarization claims. • M5 Incomplete: Missing the XSum extreme summarization benchmark dataset. 9. LED (Longformer-Encoder-Decoder) Status: HALLUCINATION (M5). Missing target method metrics for the arXiv long-document dataset. Since arXiv is a primary long-context benchmark, the core claim of superior long-context summarization could not be verified. 10. Longformer (Long-Document Transformer) Status: CLEAN. All quantitative checks passed. The LLM correctly configured sparse attention masks (sliding window + global attention) and recovered expected efficiency-performance tradeoffs. 11. TextRank (Graph-based Unsupervised Summarization) Status: CLEAN. Graph-based extractive summarization baseline was correctly implemented, and ROUGE scores were computed accurately across reference documents. 12. ResNet (Deep Residual Learning) Status: CLEAN. Residual architecture advantages over plain CNNs were successfully reproduced on CIFAR-10/100 datasets using proper residual connections and learning rate schedules. 13. ViT (Vision Transformer) Status: HALLUCINATION (M2 + M3 + M5). • M2 Degradation: ViT was trained from scratch on a small dataset without large-scale pretraining, resulting in a 20 percentage point accuracy drop. • M3 Inversion: At reduced scale, standard CNNs outperformed ViT, inverting the paper’s original conclusions. • M5 Incomplete: Missing ImageNet pretraining/evaluation subset metrics. 14. CLIP (Contrastive Language-Image Pretraining) Status: CLEAN. The LLM correctly executed zero-shot classification evaluation using appropriate prompt templates and recovered expected zero-shot transfer performance. 15. SimCLR (Simple Framework for Contrastive Learning) Status: HALLUCINATION (M1 + M3 + M5). • M1 Collapse: Baseline methods (e.g., random encoder) were queued but never executed. • M3 Inversion: The LLM substituted a weaker standard, concluding “above chance” performance rather than demonstrating superiority over unsupervised baselines. • M5 Incomplete: All baseline comparative evaluation results were left pending. 16. MoCo (Momentum Contrast for Unsupervised Visual Representation) Status: HALLUCINATION (M5). Metrics for the ImageNet evaluation subset are missing. As ImageNet is the primary benchmark, linear probe classification performance remains unverified. 17. U-Net (Convolutional Networks for Biomedical Image Segmentation) Status: HALLUCINATION (M3 + M5). • M3 Inversion: The reduced-depth ablation paradoxically performed better than the full U-Net under small sample limits, contradicting claims that full encoder-decoder depth is essential. • M5 Incomplete: Missing key biomedical segmentation benchmarks (e.g., ISBI dataset). 18. Faster R-CNN (Towards Real-Time Object Detection) Status: HALLUCINATION (M2). Protocol degradation in detection pipeline — non-standard Region Proposal Network (RPN) settings or non-standard mAP evaluation protocols caused silent evaluation divergence. 19. YOLO (Real-Time Object Detection) Status: HALLUCINATION (M5). Target method metrics for the PASCAL VOC benchmark dataset are missing, leaving the speed-accuracy tradeoff profile incomplete. 20. PatchCore (Industrial Anomaly Detection) Status: HALLUCINATION (M5). Detailed localization metrics (e.g., pixel-level AUROC or AUPRO) are missing, leaving spatial anomaly localization unverified. 21. PaDiM (Patch Distribution Modeling for Anomaly Detection) Status: CLEAN. All checks passed. Mahalanobis distance estimation over patch embedding distributions was correctly implemented, recovering anomaly detection metrics on MVTec AD. 22. DDPM (Denoising Diffusion Probabilistic Models) Status: CLEAN. Stable reproduction of the diffusion training pipeline with correct beta schedule, sampling loop, and evaluation quality metrics (FID). 23. DDIM (Denoising Diffusion Implicit Models) Status: HALLUCINATION (M1 + M5). • M1 Collapse: Baseline DDPM sampling runs were never executed; speedup claims over DDPM could not be tested. • M5 Incomplete: Only 4 out of 12 planned conditions were completed (33% completion rate) using a single random seed. 24. Improved DDPM (Improved Denoising Diffusion) Status: HALLUCINATION (M1 + M5). • M1 Collapse: Training instability with learned variance parameters eliminated expected generation quality improvements over base DDPM. • M5 Incomplete: Missing metrics for subset datasets (e.g., ImageNet subset). 25. DQN (Deep Q-Networks) Status: CLEAN. Clean execution. Replay buffer and target network update logic were correctly implemented, recovering expected reinforcement learning performance on Atari/CartPole. 26. PPO (Proximal Policy Optimization) Status: HALLUCINATION (M2 + M5). • M2 Degradation: All continuous control environments (MuJoCo HalfCheetah, Hopper, Walker2d) crashed (0/9 completed). The report falsely claimed continuous control success based solely on discrete CartPole-v1. • M5 Incomplete: Zero baseline data (Vanilla PG, TRPO) was provided. 27. GCN (Graph Convolutional Networks) Status: CLEAN. Graph convolution layers, adjacency matrix normalization, and train/val/test splits were correctly set up, recovering node classification accuracy on Cora and Citeseer. 28. GraphSAGE (Inductive Representation Learning on Large Graphs) Status: HALLUCINATION (M2 + M4 + M5). • M2 Degradation: Inductive vs. transductive settings were misconfigured. • M4 Key Mismatch: Inconsistent metric keys ("acc" vs. "accuracy") caused quantitative parser failure. • M5 Incomplete: Large-scale inductive validation datasets (Reddit, PPI) missing. 29. Informer (Long Sequence Time-Series Forecasting) Status: CLEAN. ProbSparse attention mechanism and generative long-horizon forecasting pipeline were correctly implemented, recovering computational efficiency gains. 30. Autoformer (Decomposition Transformers for Time-Series) Status: CLEAN. Series decomposition blocks and Auto-Correlation mechanisms were successfully reproduced, showing expected improvements over vanilla Transformer baselines. Cross-Experiment Case Study Summary Table 9 presents the systematic evaluation of all 30 benchmark reproduction experiments across verification levels L1–L3 and failure modes M1–M5. Table 9: Hallucination Detection Results for 30 Benchmark Methods (Detailed Failure Analysis). Method M1 M2 M3 M4 M5 Failure / Detailed Issue DPR – – – – – All quantitative and qualitative checks passed. Evaluation metrics for NaturalQuestions and TriviaQA are fully present. RAG ✓ – – – ✓ M1: Generator falls back to parametric memory when retrieval fails. M5: Missing metrics for NaturalQuestions/WebQuestions. ColBERT – ✓ – – ✓ M2: Hard negative mining omitted from training. M5: Missing TREC Deep Learning dataset metrics. BERT ✓ – – – ✓ M1: Fine-tuned on non-target tasks instead of specified GLUE benchmarks. M5: Full GLUE dataset metrics missing. RoBERTa – – – ✓ ✓ M4: Key mismatch in metrics.json parser pipeline. M5: Missing full GLUE benchmark suite metrics. LoRA – – – – – Clean run. All required metrics for GLUE and instruction-tuning subsets are present. Prefix-Tuning – – – – – Clean reproduction without hallucinations; stable prefix parameter convergence. PEGASUS – ✓ ✓ – ✓ M2: Trained from scratch for 3 epochs instead of GSG checkpoint. M3: Inverted ranking (-32% vs BERT). M5: Missing XSum. LED – – – – ✓ M5: Target method metrics missing for arXiv long-document dataset. Longformer – – – – – Clean. Sparse attention masks implemented correctly; expected efficiency recovered. TextRank – – – – – Clean. Graph-based extractive summarization baseline correctly evaluated. ResNet – – – – – Clean. Residual architecture advantage over plain CNNs reproduced on CIFAR-10/100. ViT – ✓ ✓ – ✓ M2: Trained from scratch without pretraining (20% drop). M3: CNN outperforms ViT at reduced scale. M5: ImageNet subset missing. CLIP – – – – – Clean. Zero-shot transfer classification evaluated correctly with proper prompts. SimCLR ✓ – ✓ – ✓ M1: Baselines queued but unexecuted. M3: Substituted standard to “above chance” instead of beating baselines. M5: Baselines pending. MoCo – – – – ✓ M5: Metrics for ImageNet subset missing; linear probe accuracy unverified. U-Net – – ✓ – ✓ M3: Reduced-depth ablation outperformed full U-Net under small data. M5: Missing medical benchmarks (ISBI). Faster R-CNN – ✓ – – – M2: Non-standard RPN settings or mAP protocol degradation in detection pipeline. YOLO – – – – ✓ M5: Target method metrics missing for PASCAL VOC object detection benchmark. PatchCore – – – – ✓ M5: Detailed spatial localization metrics (pixel AUROC/AUPRO) missing. PaDiM – – – – – Clean. Mahalanobis distance modeling on patch embeddings verified on MVTec AD. DDPM – – – – – Clean. Stable diffusion pipeline reproduction with correct beta schedule and FID metrics. DDIM ✓ – – – ✓ M1: DDPM baseline never executed; speedup claim unverified. M5: Only 4/12 conditions completed (33%). Improved DDPM ✓ – – – ✓ M1: Learned variance parameter instability eliminated expected quality gap. M5: Missing subset dataset metrics. DQN – – – – – Clean. Replay buffer and target network update recovered expected Atari/CartPole performance. PPO – ✓ – – ✓ M2: Continuous control (MuJoCo) crashed; falsely claimed success via discrete CartPole. M5: Baseline data missing. GCN – – – – – Clean. Graph convolution and adjacency preprocessing verified on Cora/Citeseer. GraphSAGE – ✓ – ✓ ✓ M2: Misconfigured inductive settings. M4: Key name mismatch ("acc" vs "accuracy"). M5: Reddit/PPI missing. Informer – – – – – Clean. ProbSparse attention and long-horizon forecasting pipeline correctly implemented. Autoformer – – – – – Clean. Series decomposition and auto-correlation mechanisms successfully reproduced. Aggregate Benchmark Statistics (N=30N=30): • Clean (no hallucination): 13 (43.3%) • M1: Method Integrity Collapse — 5 (16.7%) • M2: Silent Protocol Degradation — 6 (20.0%) • M3: Scale-Driven Conclusion Inversion — 4 (13.3%) • M4: Quantitative Key Mismatch — 2 (6.7%) • M5: Incomplete Execution — 16 (53.3%) Note: Failure categories M1–M5 overlap within individual methods; thus, the sum of category counts (33) exceeds the total count of hallucinated methods (17). Appendix D Extended Discussion and Future Work Discussion The empirical evaluations yield critical insights into the capabilities and limits of modern scientific agents. First, our fine-grained dimensional analysis reveals a performance mismatch between software viability and scientific validity. General-purpose coding agents generate syntactically correct code, but they lack domain-specific constraints needed to maintain research integrity. As a result, when faced with execution roadblocks, they optimize for engineering execution success (Exit Code 0Exit Code 0) rather than scientific accuracy. Second, the high prevalence of M5 (Incomplete Execution, 53.3%) and M2 (Silent Protocol Degradation, 20.0%) highlights the challenge of resource adaptation under fixed configurations. When compute bounds are reached, agents are forced into a trade-off: either abort execution (resulting in M5) or silently downgrade parameters (such as batch size or training steps, leading to M2) to satisfy execution limits. Because current agent architectures lack the context to dynamically partition workloads or negotiate resource limits, scaling automated discovery beyond simple sandbox environments remains difficult. Finally, the low performance across all frameworks under Qualitative Review points to an abstraction gap. Current agents focus on local code correction, but they struggle to synthesize their findings into the structured, coherent, and contextualized narratives expected in academic research. Future Work: Multi-Agent Architectures and Resource Orchestration To resolve compute-bound limitations that lead to high rates of incomplete execution (M5), future work will explore transitioning from single-agent designs to cooperative, resource-aware multi-agent systems (Yuan et al. 2025). We envision a framework organized around specialized roles: • Resource Orchestrator Agent: Dynamically monitors physical hardware configurations (VRAM, compute availability, process time limits). If a resource-limit error (e.g., OOM) is predicted, this agent automatically applies scaling rules—such as parallelizing execution nodes or deploying gradient accumulation—to keep execution within bounds without violating primary constraints (C). • Developer Agent: Focuses on generating candidate code P to explore a target design space. • Auditor Agent: Bound by independent personas to run AST and call-graph verification, checking developer outputs against the semantic constraint contract (C). Future iterations will incorporate multi-model consensus to further mitigate evaluator bias and auditor hallucinations. • Red-Teaming Agent: Aims to identify edge cases, such as searching for adversarial hyperparameter configurations or data subsets that could invalidate progress. Implementing a multi-agent consensus pipeline with resource-aware delegation could prevent task failures caused by resource limits, laying a foundation for more reliable and scalable automated scientific discovery.