Paper deep dive
Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Xuehai Wang, Haowei Qin, Tongxin Liu, Junkai Li, Buqiang Xu, Jintian Zhang, Yijun Chen, Zirui Xue, Shumin Deng
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly specify the analyses, methods, and success criteria required to complete the task. As a result, agents may miss important analyses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address the problem, we present AutoSciRub, an evaluation-first framework that induces a task-specific executable rubric before research execution, and uses it to guide execution, criterion-level verification as well as iterative revision. AutoSciRub decomposes an underspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria. The resulting rubric makes implicit experimental and evidential requirements explicit, providing guidance for experiments and analyses. During revision, rubric-guided verification identifies unmet criteria and enables targeted refinement of the research report and its supporting artifacts. On ResearchClawBench, AutoSciRub consistently improves all tested configurations, with an average gain of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a randomly sampled 20-task subset of AstaBench E2E Discovery, AutoSciRub further achieves an average improvement of 16.8 points across three agent harnesses, while maintaining or increasing the number of successfully completed tasks. These results demonstrate that evaluation-first guidance provides an effective and generalizable control mechanism for autonomous scientific research (Code: this https URL).
Tags
Links
- Source: https://arxiv.org/abs/2608.31076v1
- Canonical: https://arxiv.org/abs/2608.31076v1
Trouble viewing inline? Open PDF directly →
Full Text
87,964 characters extracted from source content.
Expand or collapse full text
Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents Xuehai Wang 1 , Haowei Qin 2 , Tongxin Liu 3 , Junkai Li 4 , Buqiang Xu 1 , Jintian Zhang 1 , Yijun Chen 1 , Zirui Xue 1 , Shumin Deng 1∗ 1 Zhejiang University 2 University of Electronic Science and Technology of China 3 Beijing University of Posts and Telecommunications 4 Zhejiang University of Technology 22651308@zju.edu.cn, 231sm@zju.edu.cn Abstract Autonomous scientific research agents are increasingly ap- plied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly spec- ify the analyses, methods, and success criteria required to complete the task. As a result, agents may miss important anal- yses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address the problem, we present AutoSciRub, an evaluation-first framework that in- duces a task-specific executable rubric before research execu- tion, and uses it to guide execution, criterion-level verification as well as iterative revision. AutoSciRub decomposes an un- derspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthe- sizes specific, actionable, and verifiable criteria. The resulting rubric makes implicit experimental and evidential require- ments explicit, providing guidance for experiments and analy- ses. During revision, rubric-guided verification identifies un- met criteria and enables targeted refinement of the research re- port and its supporting artifacts. On ResearchClawBench, Au- toSciRub consistently improves all tested configurations, with an average gain of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a randomly sampled 20-task subset of AstaBench E2E Dis- covery, AutoSciRub further achieves an average improvement of 16.8 points across three agent harnesses, while maintain- ing or increasing the number of successfully completed tasks. These results demonstrate that evaluation-first guidance pro- vides an effective and generalizable control mechanism for autonomous scientific research 1 . 1 Introduction Large language model agents are increasingly being used to automate scientific research workflows, including literature review, hypothesis formulation, experimental design, code implementation, result analysis, and scientific report genera- tion (Wei et al. 2025). Recent systems have further advanced toward multi-agent and long-horizon research processes that produce both scientific reports and supporting experimental 1 The code is available at https://github.com/zjunlp/AutoSciRub. ∗ Corresponding author. Open-ended Research TaskImplicit Requirements Combine diverse observations to measure global glacier mass loss... Objectives Methods Evidence Success Criteria Unsupported Conclusions Missed Analyses Wrong Procedures Final Report Planning Execution Figure 1: Why autonomous research agents struggle with open-ended research tasks. artifacts (Lu et al. 2024; Schmidgall et al. 2025; Mitchener et al. 2025). However, as their execution capabilities im- prove, a fundamental challenge becomes more pressing: how can we ensure that the resulting research actually fulfills the scientific intent of the original instruction? Scientific instruc- tions often specify only a high-level objective while leaving intermediate goals, methodological requirements, expected evidence, and success conditions implicit. As a result, an agent may produce a plausible-looking report while omit- ting essential analyses, using inappropriate procedures, or making claims unsupported by experimental evidence. Fig- ure 1 illustrates how these implicit requirements can lead to failures throughout research planning and execution. Existing scientific-agent benchmarks evaluate research outputs using executable metrics, expert-authored rubrics, and LLM-based judgments (Yehudai et al. 2026). Expert- authored rubrics are particularly useful because they provide explicit and interpretable criteria for assessing complex re- search artifacts (Starace et al. 2025). However, constructing them requires substantial domain expertise and manual ef- fort, making them difficult to scale to new research tasks. Recent work on automatic rubric generation offers a promis- ing alternative by deriving task-specific criteria from natural- language instructions and contextual information (Chen et al. 2026a; Siro, Aliannejadi, and Aliannejadi 2026; Ding 2026). Nevertheless, scientific rubric generation is especially chal- arXiv:2608.31076v1 [cs.CL] 31 Aug 2026 lenging because valid criteria depend not only on the in- struction, but also on task-relevant literature, available data, domain conventions, and execution constraints. Without such grounding, automatically generated criteria may be incom- plete, scientifically unjustified, or infeasible to verify. Moreover, rubrics in existing scientific-agent benchmarks are predominantly treated as post-hoc evaluation instru- ments. They score completed artifacts, but do not help the agent determine what evidence should be produced or how an incomplete artifact should be improved. We argue that rubrics should instead serve as intermediate scientific spec- ifications. They should make the implicit requirements of a research task explicit and connect task interpretation, re- search execution, output verification, and iterative revision. Under this view, a reliable research agent should learn to evaluate before improving. Based on this principle, we introduce AutoSciRub, a framework that automatically induces evidence-grounded scientific rubrics and uses them to guide iterative research improvement. AutoSciRub consists of two stages. First, Auto- matic Rubric Induction decomposes the original instruction into atomic scientific goals and grounds them in relevant lit- erature, web evidence, available task data, and environmental constraints. The resulting task-specific rubric makes the re- quired analyses, evidence, and success conditions explicit. Second, after the agent produces a research report and sup- porting artifacts, Rubric-Guided Iterative Revision evaluates the available evidence criterion by criterion, identifies un- met scientific requirements, and provides targeted feedback for revision. We evaluate AutoSciRub on all 40 tasks in Research- ClawBench and a fixed subset of 20 end-to-end scientific discovery tasks from AstaBench. On ResearchClawBench, AutoSciRub yields an average improvement of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three agent harnesses using the fixed DeepSeek-V4-Flash backbone. On AstaBench, it further im- proves scores by an average of 16.8 points across three agent systems. These results demonstrate that automatically in- duced, evidence-grounded rubrics can provide scalable eval- uation criteria while also serving as effective guidance for autonomous scientific research. 2 Related Work Autonomous Scientific Research Agents and Bench- marks. Early systems organize end-to-end scientific research as single-agent pipelines or centrally orchestrated work- flows, coordinating literature review, experimentation, and manuscript preparation within a unified process (Lu et al. 2024; Yamada et al. 2025; Ifargan et al. 2024). Multi-agent systems instead distribute these stages across specialized roles for collaborative ideation, critical discussion, experi- ment execution, and research coordination (Schmidgall et al. 2025; Ghareeb et al. 2026; Tang et al. 2025; Gottweis et al. 2026), while shared research states support long-horizon ex- ploration, failure recovery, and iterative hypothesis refine- ment (Mitchener et al. 2025; Lyu et al. 2026; Gao, Fang, and Zitnik 2026; Liu et al. 2026; Lu et al. 2026). Complemen- tary memory and context-management systems further sup- port persistent agents through structured long-horizon mem- ory (Xu et al. 2026a; Li et al. 2026a), cache-efficient context control (Xu et al. 2026b), and hierarchical multimodal ex- perience storage (Chen et al. 2026c). In parallel, scientific- agent benchmarks have progressed from executable coding and model-development tasks under fixed objectives (Chen et al. 2025; Chan et al. 2025), to research reproduction that requires reconstructing methods and experiments (Starace et al. 2025; Luo et al. 2026), and further to end-to-end settings that assess experiment formulation, raw-evidence analysis, and scientific report generation from high-level in- structions (Bragg et al. 2025; Garikaparthi, Patwardhan, and Cohan 2026; Wang et al. 2026; Xu et al. 2026c). Rubric-Guided Evaluation and Iterative Refinement. Rubric-based evaluation offers a structured alternative to holistic judging through explicit, fine-grained crite- ria (Hashemi et al. 2024; Chen et al. 2026a). Expert-authored hierarchical rubrics decompose complex research artifacts into independently assessable components (Starace et al. 2025; Sharma et al. 2025), while later benchmarks introduce atomic, verifiable criteria and question-specific scoring or de- duction points (Li et al. 2026b; Fan et al. 2024). Rubric con- struction has become increasingly automated (Siro, Alian- nejadi, and Aliannejadi 2026; Ding 2026; Yu et al. 2026). Recent methods generate dataset- or instance-specific cri- teria (Wang and Blanco 2026), retrieve rubric knowledge from related queries (Dhole and Agichtein 2026; Chen et al. 2026b), or recursively expand questions into structured crite- ria trees (Gao et al. 2026; Zhu et al. 2026); recursive decom- position and filtering further reduce redundancy, overlap, and preference misalignment (Shen et al. 2026; Hong et al. 2026). Recent refinement frameworks combine explicit rubrics with test-time verifiers or pre-execution checks to assess outputs against task-specific criteria (Wan et al. 2026; LeVine et al. 2026; Ye et al. 2026; Zheng et al. 2026). Criterion-level diag- noses are then fed back into generation for targeted revision of responses (Madaan et al. 2023; Gou et al. 2024), agent trajectories, and tool-use programs (LeVine et al. 2026). 3 Method Autonomous research agents often revise their outputs with- out an explicit specification of the scientific requirements that the task entails. Our framework addresses this limita- tion by first transforming an underspecified research instruc- tion into a task-specific executable rubric and then using the rubric to guide research execution, verification, and revi- sion. As illustrated in Figure 2, Automatic Rubric Induction first constructs an instruction-derived rubric skeleton through Rubric Skeleton Induction, and then transforms it into a task-specific executable rubric through Scientific Literature Grounding, Task-Data Exploration, and Criterion Synthesis. Rubric-Guided Iterative Revision subsequently evaluates the generated research artifact criterion by criterion and provides targeted feedback on missing experiments, comparisons, and evidence. We first formulate the research task and then de- scribe the two stages in detail. Rubr ic Rub ric Rubr ic Rub ric Rubr ic Rub ric Goal: to ...... DataSources: Experiments: Metrics: Artifacts: Rubric I. Automatic Rubric Induction query 1 goal 1 goal 2 goal 3 query 2 query 3 Goal 3 Goal 2 Goals Rubric Rubric Rubric1 Goal: to ...... DataSources: Expreiments: Metrics: Artifacts: Rubric Rubric Rubric1 Goal: to ...... DataSources: Expreiments: Metrics: Artifacts: Rubric Rubric Rubric Goal: to ...... DataSources: Experiments: Metrics: Artifacts: I. Rubric-Guided Iterative Revision Execution Verification Rubric Missing experiment Incomplete comparison Feedback Final Report Input Grounded Knowledge Format Fields DataRelations Data Profile ... Scientific Literature Grounding 2 Task-Data Exploration 3 Criterion Synthesis 4 Rubric Skeleton Induction 1 Filtering papers data docs method docs Search tool set Instruction Methods & Protocols Metrics & Baselines Controls & Ablations Evidence Artifacts Chem.Earth Info. Energy MathNeuro. PhysicsAstro. Data ... Figure 2: Overview of AutoSciRub. (1) Automatic Rubric Induction: starting from a multi-domain scientific task, AutoSciRub induces a rubric skeleton, grounds it in scientific literature, explores the task-visible data, and synthesizes a task-specific executable rubric. (2) Rubric-Guided Iterative Revision: the induced rubric guides research execution and supports an iterative feedback loop that verifies and improves the generated artifact into the final report. 3.1 Task Formulation We consider a collection of multi-domain, end-to-end au- tonomous scientific research tasks D spanning Astronomy, Chemistry, Earth Science, Energy Science, Information Sci- ence, Life Science, Materials Science, Mathematics, Neu- roscience, and Physics. Each task is represented as τ i = (x i ,E i ), where x i is a high-level research instruction andE i is the task-visible environment, including available literature, data, search, code execution, and domain-specific tools. Given τ i , a research agent π performs scientific reasoning, experimentation, and analysis to produce A π i = π(x i ,E i ) = (r i ,S i ),(1) wherer i is the scientific report andS i contains its supporting code, results, analyses, tables, and figures. Research instructions are typically underspecified: they state a broad objective without fully defining the scientific questions, comparisons, or evidence required. We therefore associate each task with a latent scientific specification Z ∗ i = (G ∗ i ,C ∗ i ),(2) where G ∗ i denotes the required scientific goals and C ∗ i the evidence requirements for verifying them. Artifact quality is defined by q π i = Eval (A π i ;Z ∗ i ),(3) where Eval(·) assesses goal coverage and evidence adequacy. Our objective is to improve the artifact quality across tasks. Since neitherZ ∗ i nor its evaluator is available during ex- ecution, AutoSciRub induces an instruction-derived rubric skeleton b G i and a task-specific executable rubric b R i from the instruction, scientific literature, and task-visible data. The rubric operationalizes the latent specification and guides ex- ecution, criterion-level verification, and iterative revision. 3.2 Automatic Rubric Induction Rather than selecting criteria from a fixed library, we induce a task-specific executable rubric at inference time. As shown in Figure 2, this process consists of four steps: Rubric Skele- ton Induction, Scientific Literature Grounding, Task-Data Exploration, and Criterion Synthesis. These steps make the instruction’s implicit requirements explicit, ground them in established scientific practice and task-visible resources, and convert them into verifiable criteria. Rubric Skeleton Induction. A high-level research instruc- tion often implies multiple analyses, comparisons, and inter- pretations without listing them explicitly. We organize these requirements into a compact set of atomic scientific goals: b G i = φ inst (x i ) =g i,k K i k=1 ,(4) where K i is the number of induced goals and g i,k = (n i,k ,s i,k ) contains a concise goal name n i,k and a con- crete scientific requirement s i,k . Together, these goals form an instruction-derived rubric skeleton. The goals must be traceable to the instruction, cover its main requirements, and avoid unnecessary overlap. At this stage, the agent observes only x i and does not introduce specific methods, metrics, baselines, or expected results. The skeleton therefore defines what the task should address, while later steps determine how each goal should be established. Scientific Literature Grounding. The rubric skeleton de- fines the task scope but not the scientific practice needed to satisfy each goal. For every g i,k , the agent forms broad queries over relevant concepts, methods, metrics, and stan- dard protocols. It first consults task-provided related litera- ture and then supplements missing coverage through the na- tive web-search capability of the agent harness and external services, including arXiv, OpenAlex, Semantic Scholar, and Tavily. The retrieved sources may include papers, preprints, official dataset or benchmark pages, method repositories, and software documentation. Queries avoid long instruction spans, task identifiers, and task-specific filenames. Candidate papers are ranked by rel- evance to the induced goals. Before full-text access, the re- trieval layer removes candidates whose titles match a hidden target-paper blocklist maintained by the evaluation harness and unavailable to the induction agent. The agent then retains five to seven core papers for the task and consults additional documentation when needed. For each goal, the selected sources are summarized as K i,k = φ lit g i,k ; e L i ,(5) where e L i is the filtered source set. K i,k records relevant methods and protocols, typical analyses, metrics, baselines, controls, ablations, robustness checks, and useful evidence forms. The information is organized by scientific goal rather than by individual paper. Retrieved sources guide experi- ment and evidence design but are not treated as experimental evidence for the current task. Task-Data Exploration. Literature grounding describes what is scientifically appropriate, but the proposed analy- ses must also be supported by the available resources. The agent therefore performs a lightweight inspection of the task- visible data: P i = φ data (E i ),(6) where P i records the available files and datasets, their for- mats and dimensions, key fields, units, labels, experimental conditions, cross-source relations, and relevant constraints. This profile identifies which data sources support each goal and which literature-suggested analyses are feasible without assuming unavailable labels, conditions, or reference values. It is used only for planning; full experiments and analyses are deferred to task execution. Criterion Synthesis. Finally, the agent combines the rubric skeleton, literature-grounded knowledge, and task- data profile. It selects feasible experiments and analyses for each goal and specifies the evidence needed to support the resulting claims. These requirements are synthesized into a task-specific executable rubric: b R i = φ syn b G i ,K i,k K i k=1 ,P i =ρ i,j M i j=1 , (7) where M i is the number of synthesized criteria. Each crite- rionρ i,j links to one or more scientific goals and specifies the relevant data sources, required experiments or analyses, eval- uation metrics and comparisons, expected evidence artifacts, and its satisfaction condition. The resulting rubric aligns established scientific practice with the evidence that can be produced from the current task, providing an explicit specification for subsequent research execution and verification. 3.3 Rubric-Guided Iterative Revision The induced rubric is provided to the backbone research agent as an execution-time specification. Conditioned on b R i , the agent performs the required experiments and analyses and produces an initial artifact A (0) i = (r (0) i ,S (0) i ), containing the report and its supporting code, results, tables, and figures. Claude CodeCodexOpenClaw 50 60 70 80 90 100 Mean Score Vanilla+ AutoSciRub 71.26 71.74 67.96 90.62 84.35 86.34 +19.4 +12.6 +18.4 18 19 20 Successful Tasks 18/2018/20 19/20 20/20 20/20 19/20 Figure 3: Performance comparison on AstaBench. Au- toSciRub consistently improves mean benchmark scores across all three evaluated agent configurations. Criterion-Level Verification. The initial execution may still leave some rubric requirements unmet. At revision round t, a verifier checks each criterion against the current artifact: z (t) i,j ,d (t) i,j = Verify ρ i,j ,A (t) i ,(8) where z (t) i,j ∈ 0, 1 indicates whether the criterion is sat- isfied, and d (t) i,j describes the remaining evidence gap. The verifier considers the required experiments, reported results, supporting artifacts, and whether the conclusions are ade- quately supported. This criterion-level check identifies spe- cific omissions rather than returning only a holistic score. Targeted Revision. Failed criteria and their diagnoses form the revision feedback ∆ (t) i . The agent then updates the current artifact by addressing these gaps: A (t+1) i = π rev A (t) i , ∆ (t) i ,E i | b R i .(9) A revision may add missing experiments or comparisons, correct evidence artifacts, strengthen the analysis, or remove unsupported claims. The process stops when all criteria are satisfied or the revision budget is reached, yielding the final artifactA ∗ i =A (T) i . 4 Experiments 4.1 Experimental Setup Benchmarks and Metrics. We evaluate AutoSciRub on ResearchClawBench (Xu et al. 2026c) and the End-to-End Discovery category of AstaBench (Bragg et al. 2025). Re- searchClawBench contains 40 tasks across ten scientific do- mains. AstaBench evaluates end-to-end AI and NLP research from experimentation to report generation, and we randomly sample 20 tasks from its Easy split. We report the average rubric-based score on both benchmarks and task completion on AstaBench. Further details are provided in Appendix A. Implementation Details. We compare each vanilla agent with its AutoSciRub-augmented counterpart under identi- cal task inputs, tools, and execution environments. For Re- searchClawBench, we evaluate both cross-model and cross- harness generalization. In cross-model evaluation, we fix A. Cross-Model Generalization Fixed agent harness:Codex Backbone LLMSettingOverallAstro.Chem.EarthEnergyInfo.LifeMater.MathNeuro.Phys. vanilla18.6629.6910.9721.2517.8717.6116.0217.5516.655.6033.43 GPT-5.4 +AutoSciRub21.04↑ 2.3831.43↑ 1.7411.62↑ 0.6522.22↑ 0.9722.70↑ 4.8313.92↓ 3.6919.85↑ 3.8322.70↑ 5.1520.57↑ 3.927.72↑ 2.1237.69↑ 4.26 vanilla20.8631.5611.9623.9917.9515.6916.8822.1424.205.8438.40 GLM-5.2 +AutoSciRub22.73↑ 1.8731.99↑ 0.4317.85↑ 5.8923.25↓ 0.7420.70↑ 2.7518.00↑ 2.3117.34↑ 0.4623.48↑ 1.3423.03↓ 1.1712.58↑ 6.7439.06↑ 0.66 vanilla19.0528.0113.3619.1419.659.3115.1022.3122.966.3034.37 MiniMax-M3 +AutoSciRub21.04↑ 1.9927.10↓ 0.9113.94↑ 0.5821.53↑ 2.3926.46↑ 6.8112.20↑ 2.8917.58↑ 2.4820.86↓ 1.4522.38↓ 0.5812.25↑ 5.9536.10↑ 1.73 B. Cross-Harness Generalization Fixed backbone LLM:DeepSeek-V4-Flash Agent HarnessSettingOverallAstro.Chem.EarthEnergyInfo.LifeMater.MathNeuro.Phys. vanilla16.6028.359.3915.3224.609.2713.5720.0912.644.0428.70 Claude Code +AutoSciRub18.74↑ 2.1428.15↓ 0.209.48↑ 0.0920.30↑ 4.9824.75↑ 0.1511.06↑ 1.7912.79↓ 0.7821.30↑ 1.2117.91↑ 5.277.67↑ 3.6333.95↑ 5.25 vanilla17.2524.268.9022.3918.939.2915.7019.0715.085.1133.71 OpenClaw +AutoSciRub20.36↑ 3.1130.31↑ 6.0512.59↑ 3.6822.85↑ 0.4525.24↑ 6.3216.20↑ 6.9017.18↑ 1.4717.95↓ 1.1323.12↑ 8.038.56↑ 3.4529.62↓ 4.09 vanilla14.4921.129.5915.2016.9610.689.2920.2812.057.0522.73 OpenScience +AutoSciRub18.09↑ 3.6026.58↑ 5.4612.04↑ 2.4519.34↑ 4.1422.94↑ 5.9813.82↑ 3.1417.53↑ 8.2416.02↓ 4.2617.45↑ 5.407.06↑ 0.0128.11↑ 5.38 Table 1: Main results on ResearchClawBench. Panel A evaluates generalization across backbone LLMs under the fixed Codex harness, while Panel B evaluates generalization across agent harnesses using the fixed DeepSeek-V4-Flash backbone. Vanilla denotes the unmodified configuration, and +AutoSciRub denotes the same configuration augmented with our plugin layer. Bold values indicate the better result within each paired comparison. Green and red arrows denote absolute improvements and degradations, respectively. Codex (OpenAI 2026a) as the agent harness and use GPT- 5.4 (OpenAI 2026b), GLM-5.2 (Z.ai 2026), and MiniMax- M3 (MiniMax 2026) as backbone models. In cross-harness evaluation, we fix DeepSeek-V4-Flash (DeepSeek-AI 2026) as the backbone model and compare Claude Code (An- thropic 2026), OpenClaw (OpenClaw Foundation 2026), and OpenScience (Synthetic Sciences 2026). Following the benchmark-specific evaluation protocols, we use GPT- 5.1 (OpenAI 2025) to evaluate report quality. For AstaBench, we evaluate three representative agent con- figurations: Claude Code with DeepSeek-V4-Flash, Open- Claw with DeepSeek-V4-Flash, and Codex with GPT-5.4- mini. The generated artifacts are evaluated using MiniMax- M3 as the evaluator. Across both benchmarks, each resulting submission is independently scored three times, and the mean score is reported. Additional implementation details, includ- ing agent configurations, prompts, and execution settings, are provided in Appendix A. 4.2 Overall Performance Results on ResearchClawBench. As shown in Table 1, AutoSciRub improves the overall ResearchClawBench score in all six model–harness configurations. Under the fixed Codex harness, AutoSciRub yields gains of 2.38 points for GPT-5.4, 1.87 points for GLM-5.2, and 1.99 points for MiniMax-M3. These gains remain consistent across model families and baseline strengths. Among these configurations, GLM-5.2 with AutoSciRub achieves the highest overall score of 22.73. Together, the consistent gains across three backbone LLMs indicate that the effectiveness of AutoSciRub is not model-specific. The improvements are similarly preserved when varying the agent harness. With DeepSeek-V4-Flash as the shared backbone, AutoSciRub yields gains of 2.14 points for Claude Code, 3.11 points for OpenClaw, and 3.60 points for Open- Science. The larger OpenClaw and OpenScience gains fur- ther show that AutoSciRub remains effective across agent systems with distinct workflows and baseline capabilities. At the domain level, AutoSciRub improves 49 of 60 paired com- parisons, including consistent gains in chemistry, energy sci- ence, and neuroscience across all six configurations. Overall, the improvements across backbone models, agent harnesses, and scientific domains demonstrate that AutoSciRub gener- alizes well to heterogeneous scientific research agents. Generalization to AstaBench. We further evaluate whether these benefits transfer to a different task collection and evaluation protocol. As shown in Figure 3, AutoSciRub substantially improves the mean scores of all three evaluated agents on the fixed 20-task AstaBench End-to-End Discovery subset. It yields gains of 19.36 points for Claude Code, 12.61 points for Codex, and 18.38 points for OpenClaw, corre- sponding to an average improvement of 16.78 points. Claude Code and Codex also increase their successful-task counts from 18/20 to 20/20. OpenClaw completes 19/20 tasks under both settings while gaining 18.38 points, showing that Au- toSciRub improves the quality of completed research outputs in addition to task completion. 4.3 Ablation Study Table 2 presents a cumulative stage-wise ablation on all 40 ResearchClawBench tasks using OpenClaw with DeepSeek- V4-Flash. Starting from the vanilla agent, Rubric Skeleton Induction (Skel.) increases the score from 17.25 to 17.61. This modest gain suggests that making the instruction’s im- plicit scientific requirements explicit helps the agent identify what should be investigated, but offers limited guidance on how these requirements should be carried out and verified. Specificity Actionability Scientific Core Coverage Evidence Verifiability 1 3 5 Rubric SkeletonExecutable Rubric (a) Rubric quality 12345 Scientific Core Coverage 0 10 20 30 40 Report score slope = 9.94 Rubric Skeleton r = 0.63 ρ = 0.63 R 2 = 0.39 12345 Scientific Core Coverage slope = 6.80 Executable Rubric r = 0.55 ρ = 0.59 R 2 = 0.31 (b) Scientific core coverage vs. report score. 50403020100 01020304050 Score gain Score loss Life003 Info.004 Earth002 Math003 Earth003 Energy002 Chem.004 Chem.003 Math001 Chem.001 Astro.002 Chem.002 Mater.001 Mater.002 Neuro.002 Energy003 Energy001 Earth004 Astro.001 Phys.003 Astro.003 Astro.004 Info.002 Math004 Life001 Phys.004 Info.001 Energy004 Neuro.001 Life002 Neuro.004 Neuro.003 Info.003 Phys.002 Life004 Mater.004 Earth001 Math002 Phys.001 Mater.003 (c) Task-level score changes. Figure 4: Rubric quality and its downstream effects. (a) Mean quality profiles of rubric skeletons and executable rubrics across 40 tasks. (b) Association between scientific core coverage and downstream report score. (c) Task-level report-score changes from rubric skeletons to executable rubrics. goal: "Identify and characterize Floquet-Bloch replica bands, including gap measurements at avoided crossings.", experiments: "Extract EDCs at Γ point and at avoided-crossing momenta", "Fit multi-peak Voigt profiles to EDCs", "..." required_artifacts: “type”: “figure”, “must_show”: “EDC fits revealing n = -1, 0, and +1 sidebands”, “description_requirement”: “Demonstrate sidebands with peak separation matching ħω ≈ 248 meV ...”,"..." metrics: "ħω from peak separation (~248 meV)", "..." baselines: "Compare measured gap 2Δ with Floquet theory", "..." data_sources: “data/raw_trARPES_data.h5”,"..." 2/5 Executable Rubric content: "Energy-momentum map from tr- ARPES showing the main Dirac cone and a clear replica band induced by the 5 μm pump excitation." keywords •Clear visualization of the main Dirac cone. •Clear visualization of a replica band shifted from the main cone •Axes labeled with energy (eV) and momentum (Å⁻¹ ). •Data acquired at a time delay of ~1 ps after pump excitation 1/3 Golden Checklist Input: Monolayer epitaxial graphene samples and mid-infrared pump excitation parameters (wavelength: 5 μm, intensity, polarization angle). Output: Direct, energy-and momentum-resolved observation of Floquet-Bloch states (replica bands of the Dirac cone) via time-resolved and angle-resolved photoemission spectroscopy (tr-ARPES). Scientific Goal: To experimentally confirm the existence of Floquet-Bloch states in a paradigmatic 2D material and elucidate the underlying scattering mechanism involving photon- dressed Volkov final states. TASK Target Paper Report Figure 5: Comparison between an AutoSciRub executable rubric and the benchmark-provided golden checklist. Adding the grounded-rubric stage (Grd.) further raises the score to 18.31, corresponding to a gain of 1.06 points over the base and an additional 0.70 points over skeleton induction alone. By grounding the induced goals in scientific literature, examining the task-visible data, and synthesizing feasible experiments and evidence requirements, this stage turns a high-level rubric skeleton into more concrete and verifiable criteria. Rubric-Guided Iterative Revision (Rev.) achieves the best score of 20.36, improving by 3.11 over the base and 2.05 over the grounded-rubric setting. This largest incremental gain indicates that the rubric is most effective when it is used not only to guide execution, but also to identify unmet criteria and support targeted revision of the report and its supporting artifacts. Overall, the monotonic gains show that rubric skeleton induction, grounding, and iterative revision play complementary roles in turning underspecified research instructions into better-supported research outputs. SettingSkel. Grd. Rev. Score↑∆ Base Base✗ 17.25– w/ Skeleton✓✗ 17.61 +0.36 w/ Grounded Rubric✓✗ 18.31 +1.06 Full✓ 20.36 +3.11 Table 2: Stage-wise cumulative ablation on ResearchClaw- Bench. Skel. denotes Rubric Skeleton Induction; Grd. adds Scientific Literature Grounding, Task-Data Exploration, and Criterion Synthesis; and Rev. adds Rubric-Guided Iterative Revision. Scores are averaged over the same 40 tasks, and ∆ Base denotes the improvement over the base setting. 4.4 Analysis Rubric Induction Produces Executable Guidance. Open-ended research instructions rarely specify the experi- ments, evidence, and success conditions needed to complete a task. Figure 5 shows how AutoSciRub addresses this un- derspecification. While the benchmark checklist mainly de- scribes what should appear in the final report, the executable rubric also specifies how the required evidence should be produced and verified through concrete experiments, arti- facts, metrics, baselines, and data sources. It therefore serves as an execution-time specification rather than only a post-hoc evaluation instrument. For each of the 40 ResearchClawBench tasks, we compare the instruction-derived rubric skeleton with the task-specific executable rubric produced after literature grounding, task- data exploration, and criterion synthesis. As shown in Fig- ure 4a, the mean score across the four evaluated dimensions increases from 2.20 to 3.84. The largest improvement occurs in specificity, which increases from 1.65 to 4.40 (+2.75), followed by evidence verifiability from 1.78 to 4.08 (+2.30) and actionability from 2.00 to 3.83 (+1.83). These improve- ments show that rubric induction effectively operationalizes broad scientific goals into concrete, actionable, and verifiable requirements for research execution and verification. InitialRevision 1Revision 2Revision 3 0 5 10 15 20 25 30 Number of tasks 23/23 11/11 1/6 1 passed 14 15 16 17 18 19 20 21 22 23 24 25 Mean score Stopped tasks Verifier-passed tasks Overall Stop@R1 Stop@R2 Stop@R3 Held after stop Stopping point 18.31 19.47 20.08 20.36 20.47 21.92 14.16 15.12 17.71 17.63 18.04 17.34 19.23 (a) Overall refinement dynamics. -3-2-10+1+2+3+4+5+6+7 0 1 2 3 4 5 6 7 8 9 10 11 Number of tasks 1 2 1 10 99 222 1 +14+15 1 Improved tasks (n=36) Declined tasks (n=4) (b) Distribution of task-level score changes. -10+1+2+3+4+5 Math Earth Info. Life Chem. Astro. Neuro. Energy Mater. Phys +4.12 +3.73 +3.32 +2.00 +1.93 +1.92 +1.69 +1.43 +0.48 -0.11 Improved domains Declined domains (c) Domain-level mean improvement. Figure 6: Analysis of rubric-guided refinement. (a) Overall score and task stopping behavior across refinement steps. (b) Distribution of score changes from the initial reports to the final reports. (c) Mean score improvement across scientific domains. Revision Round Rubric-Free Self-RefinementRubric-Guided Revision Score∆Score∆ 118.80+0.4919.47+1.16 218.52+0.2120.08+1.77 319.08+0.7720.36+2.05 Table 3: Comparison of rubric-free self-refinement and rubric-guided revision across three revision rounds on all 40 ResearchClawBench tasks. ∆ values denote score changes relative to the shared unrevised baseline score of 18.31. Rubric Induction Improves Operationalization but Not Scientific Framing. Although rubric induction substan- tially improves specificity, actionability, and evidence verifi- ability, scientific core coverage slightly decreases from 3.35 to 3.07. This contrast suggests that AutoSciRub is more ef- fective at translating an identified scientific direction into concrete and verifiable requirements than at revising the un- derlying framing of the research problem. As shown in Figure 4b, scientific core coverage remains positively associated with report quality for both rubric skele- tons and executable rubrics. Figure 4c further shows that executable rubrics yield a mean report-score gain of 0.70 across all 40 tasks, with improvements observed on 34 tasks. These results suggest that AutoSciRub is effective at trans- lating an identified scientific core into concrete experiments and verifiable evidence requirements, but is less effective at identifying that core when it is missing from the initial rubric skeleton. In such cases, grounding and synthesis may elabo- rate secondary or off-target analyses rather than correct the research direction. Higher-level scientific judgment therefore remains largely dependent on the backbone model. Rubric-Guided Revision Outperforms Generic Self- Refinement. We analyze whether the improvement from iterative revision is caused by rubric guidance or merely by giving the agent additional opportunities to rewrite its re- port. Starting from the same checkpoint-0 reports, we com- pare rubric-guided revision with a rubric-free self-refinement baseline, which asks the agent to inspect and improve its pre- vious output without criterion-level feedback. As shown in Table 3, rubric-free refinement increases the average score from 18.31 to only 19.08 after three rounds and even causes a regression at checkpoint 2. In contrast, rubric-guided revi- sion improves the score monotonically to 20.36, yielding a cumulative improvement of 2.05, compared with only 0.77 without rubrics. Thus, rubric-guided revision achieves ap- proximately 2.7 times the cumulative improvement of rubric- free self-refinement. This result shows that the gain does not come from repeated rewriting alone: explicit criteria identify concrete deficiencies and direct the agent toward unsatisfied scientific requirements, whereas generic self-refinement pro- duces less stable improvements. Figure 6a further shows how the guided improvement ac- cumulates across revision rounds. The first revision raises the average score from 18.31 to 19.47, contributing an im- provement of 1.16. The second and third rounds add im- provements of 0.61 and 0.28, respectively. This decrease in aggregate gain mainly results from adaptive stopping rather than ineffective later revisions. After the first round, 23 tasks have already passed the verifier, leaving only 17 tasks for a second revision and six for a third. For these remaining tasks, the average per-round improvements are still 1.43 and 1.89, respectively. Overall, 35 of the 40 tasks pass within three revisions. These results suggest that one revision offers a strong default cost/performance trade-off, while additional rounds provide targeted improvements for harder tasks. The task- and domain-level results in Figures 6b and 6c show that the overall gain is broadly distributed rather than dominated by a few high-improvement cases. Specifically, 36 of the 40 reports improve over their initial versions, and pos- itive average gains are observed in most scientific domains. Mathematics, earth science, and information science achieve the largest average improvements of 4.12, 3.73, and 3.32, re- spectively. These results demonstrate that the improvement of rubric-guided revision is consistent across tasks and do- mains, rather than being driven by a single high-scoring case. 5 Conclusion We introduced AutoSciRub, a general framework that au- tomatically induces task-specific scientific rubrics and uses them as execution-time guidance for autonomous research agents. AutoSciRub decomposes underspecified research in- structions into scientific goals, grounds evaluation criteria in external evidence, and guides criterion-level verification and revision of generated research artifacts. Experiments on ResearchClawBench and AstaBench show that AutoSciRub consistently improves different backbone models and agent harnesses, demonstrating its effectiveness and generalizabil- ity as a plugin layer for autonomous scientific research. References Anthropic. 2026. Claude Code. https://github.com/ anthropics/claude-code. Bragg, J.; D’Arcy, M.; Balepur, N.; Bareket, D.; Dalvi, B.; et al. 2025. AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite. CoRR, abs/2510.21652. Chan, J. S.; Chowdhury, N.; Jaffe, O.; Aung, J.; Sherburn, D.; et al. 2025. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. In The Thir- teenth International Conference on Learning Representa- tions, ICLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net. Chen, H.; Han, Z.; Yan, Y.; Zhu, Q.; Sun, M.; and Che, W. 2026a. From Holistic Evaluation to Structured Crite- ria: Rubrics Across the Evolving LLM Landscape. CoRR, abs/2606.08625. Chen, Y.; Maiga, A.; Rahmani, H. A.; and Yilmaz, E. 2026b. Automated Rubrics for Reliable Evaluation of Medical Dia- logue Systems. arXiv preprint arXiv:2601.15161. Chen, Y.; Xiao, B.; Zhao, Y.; Xia, H.; Xu, B.; Fang, J.; Li, Y.; Zheng, Y.; Wang, X.; Xue, Z.; Zhang, L.; Li, H.; and Zhang, N. 2026c. LightMem-Ego: Your AI Memory for Everyday Life. CoRR, abs/2607.11487. Chen, Z.; Chen, S.; Ning, Y.; Zhang, Q.; Wang, B.; et al. 2025. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. In The Thirteenth International Conference on Learning Repre- sentations, ICLR 2025, Singapore, April 24-28, 2025. Open- Review.net. DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. CoRR, abs/2606.19348. Dhole, K. D.; and Agichtein, E. 2026. RubricRAG: To- wards Interpretable and Reliable LLM Evaluation via Do- main Knowledge Retrieval for Rubric Generation. In Moffat, A.; Scholer, F.; Bast, H.; Najork, M.; and Zhang, M., eds., Proceedings of the 49th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, SIGIR 2026, Melbourne, VIC, Australia, July 20–24, 2026, 3681–3687. ACM. Ding, L. 2026. AdaRubric: Task-Adaptive Rubrics for LLM Agent Evaluation. CoRR, abs/2603.21362. Fan, Z.; Wang, W.; Wu, X.; and Zhang, D. 2024. SedarEval: Automated Evaluation using Self-Adaptive Rubrics. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y., eds., Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, volume EMNLP 2024 of Findings of ACL, 16916–16930. Associa- tion for Computational Linguistics. Gao, S.; Fang, A.; and Zitnik, M. 2026. AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation. CoRR, abs/2605.28655. Gao, S.; Su, Y.; Sui, P.; Ginder, C.; and Zitnik, M. 2026. Qworld: Question-Specific Evaluation Criteria for LLMs. CoRR, abs/2603.23522. Garikaparthi, A.; Patwardhan, M.; and Cohan, A. 2026. Re- searchGym: Evaluating Language Model Agents on Real- World AI Research. CoRR, abs/2602.15112. Ghareeb, A. E.; Chang, B.; Mitchener, L.; Yiu, A.; Szostkiewicz, C. J.; et al. 2026. A Multi-Agent System for Automating Scientific Discovery. Nature, 655: 497–505. Gottweis, J.; Weng, W.-H.; Daryin, A.; Tu, T.; Sirkovic, P.; et al. 2026. Accelerating Scientific Discovery with Co- Scientist. Nature, 655: 487–496. Gou, Z.; Shao, Z.; Gong, Y.; Shen, Y.; Yang, Y.; Duan, N.; and Chen, W. 2024. CRITIC: Large Language Models Can Self- Correct with Tool-Interactive Critiquing. In The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Hashemi, H.; Eisner, J.; Rosset, C.; Durme, B. V.; and Kedzie, C. 2024. LLM-Rubric: A Multidimensional, Calibrated Ap- proach to Automated Evaluation of Natural Language Texts. In Ku, L.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, 13806–13834. As- sociation for Computational Linguistics. Hong, H.; Li, Y.; Chen, J.; Huy, L. G.; Ananiadou, S.; Kim, J.-j.; and Lin, C. 2026. Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction. arXiv preprint arXiv:2607.12835. Ifargan, T.; Hafner, L.; Kern, M.; Alcalay, O.; and Kishony, R. 2024. Autonomous LLM-driven research from data to human-verifiable research papers. CoRR, abs/2404.17605. LeVine, W.; Evers, B.; Saltwick, S.; and Venkatesh, A. 2026. RubricRefine: Improving Tool-Use Agent Reliabil- ity with Training-Free Pre-Execution Refinement. CoRR, abs/2605.09730. Li, D.; Liu, Z.; Wang, J.; Huang, J.; Li, F.; Jia, B.; Hu, B.; and Zhang, M. 2026a. LycheeMemory V2: Efficient Long- Term Memory for LLM Agents via Semantic Segment-Level Consolidation. arXiv:2608.12990. Li, R.; Du, M.; Xu, B.; Zhu, C.; Wang, X.; et al. 2026b. DeepResearch Bench I: Diagnosing Deep Research Agents via Rubrics from Expert Report. CoRR, abs/2601.08536. Liu, J.; Qiu, S.; Li, M.; Li, B.; Ji, H.; et al. 2026. AutoRe- searchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration. CoRR, abs/2605.20025. Lu, C.; Lu, C.; Lange, R. T.; Foerster, J. N.; Clune, J.; et al. 2024. The AI Scientist: Towards Fully Automated Open- Ended Scientific Discovery. CoRR, abs/2408.06292. Lu, Z.; Li, D.; Shi, Y.; Wang, B.; Wang, L.; and Hu, B. 2026. Structured Episodic Event Memory. In Liakata, M.; Mor- eira, V. P.; Zhang, J.; and Jurgens, D., eds., Proceedings of the 64th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, 6125–6141. Association for Computational Linguistics. Luo, Y.; Yu, Z.; Wang, X.; Zhu, Y.; Zhang, N.; Wei, L.; Du, L.; Zheng, D.; and Chen, H. 2026. What Makes AI Research Replicable? Executable Knowledge Graphs as Sci- entific Knowledge Representations. In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds., Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, 841–861. Associ- ation for Computational Linguistics. Lyu, Y.; Zhang, X.; Yi, X.; Zhao, Y.; Guo, S.; et al. 2026. EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery. CoRR, abs/2603.08127. Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self-Refine: Iterative Refinement with Self-Feedback. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems 36: An- nual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. MiniMax. 2026. MiniMax M3: Frontier Coding, 1M Con- text, Native Multimodality—All in One Model. https: //w.minimax.io/blog/minimax-m3. Accessed: 2026-07- 24. Mitchener, L.; Yiu, A.; Chang, B.; Bourdenx, M.; Nadolski, T.; et al. 2025. Kosmos: An AI Scientist for Autonomous Discovery. CoRR, abs/2511.02824. OpenAI. 2025. GPT-5.1: A Smarter, More Conversational ChatGPT. https://openai.com/index/gpt-5-1/. Accessed: 2026-07-24. OpenAI. 2026a. Codex. https://github.com/openai/codex. OpenAI. 2026b. Introducing GPT-5.4. https://openai.com/ index/introducing-gpt-5-4/. Accessed: 2026-07-24. OpenClaw Foundation. 2026. OpenClaw. https://github. com/openclaw/openclaw. Schmidgall, S.; Su, Y.; Wang, Z.; Sun, X.; Wu, J.; et al. 2025. Agent Laboratory: Using LLM Agents as Research Assis- tants. In Christodoulopoulos, C.; Chakraborty, T.; Rose, C.; and Peng, V., eds., Findings of the Association for Computa- tional Linguistics: EMNLP 2025, Suzhou, China, November 4-9, 2025, 5977–6043. Association for Computational Lin- guistics. Sharma, M.; Zhang, C. B. C.; Bandi, C.; Wang, C.; Aich, A.; Nghiem, H.; Rabbani, T.; Htet, Y.; Jang, B.; Basu, S.; Balwani, A.; Peskoff, D.; Ayestaran, M.; Hendryx, S. M.; Kenstler, B.; and Liu, B. 2025. ResearchRubrics: A Bench- mark of Prompts and Rubrics For Evaluating Deep Research Agents. CoRR, abs/2511.07685. Shen, W. F.; Qiu, X.; Whitehouse, C.; Alazraki, L.; Goel, S.; et al. 2026. Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks. CoRR, abs/2602.05125. Siro, C.; Aliannejadi, P.; and Aliannejadi, M. 2026. Learn- ing to Judge: LLMs Designing and Applying Evaluation Rubrics. In Demberg, V.; Inui, K.; and Marquez, L., eds., Findings of the Association for Computational Linguistics: EACL 2026, Rabat, Morocco, March 24-29, 2026, Findings of ACL, 6371–6389. Association for Computational Linguis- tics. Starace, G.; Jaffe, O.; Sherburn, D.; Aung, J.; Chan, J. S.; et al. 2025. PaperBench: Evaluating AI’s Ability to Replicate AI Research. In Singh, A.; Fazel, M.; Hsu, D.; Lacoste-Julien, S.; Berkenkamp, F.; Maharaj, T.; Wagstaff, K.; and Zhu, J., eds., Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13- 19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net. Synthetic Sciences. 2026. OpenScience: The Open-Source AI Workbench for Scientific Research. https://github.com/ synthetic-sciences/openscience. Tang, J.; Xia, L.; Li, Z.; and Huang, C. 2025. AI-Researcher: Autonomous Scientific Innovation. CoRR, abs/2505.18705. Wan, Y.; Fang, T.; Li, Z.; Huo, Y.; Wang, W.; Mi, H.; Yu, D.; and Lyu, M. R. 2026. Inference-Time Scaling of Verifi- cation: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification. In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds., Findings of the Associa- tion for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2-7, 2026, 24822–24835. As- sociation for Computational Linguistics. Wang, Z.; Bai, F.; Luo, Z.; Su, J.; Sun, K.; Yu, X.; Liu, J.; Zhou, K.; Cardie, C.; Dredze, M.; Xing, E. P.; and Hu, Z. 2026. FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights. CoRR, abs/2602.02905. Wang, Z.; and Blanco, E. 2026. Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge. CoRR, abs/2605.30568. Wei, J.; Yang, Y.; Zhang, X.; Chen, Y.; Zhuang, X.; Gao, Z.; Zhou, D.; Wang, G.; Gao, Z.; Cao, J.; Qiu, Z.; He, X.; Zhang, Q.; You, C.; Zheng, S.; Ding, N.; Ouyang, W.; Dong, N.; Cheng, Y.; Sun, S.; Bai, L.; and Zhou, B. 2025. From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery. CoRR, abs/2508.14111. Xu, B.; Chen, Y.; Fang, J.; Zhong, R.; Yao, Y.; Zhu, Y.; Du, L.; and Deng, S. 2026a. StructMem: Structured Memory for Long-Horizon Behavior in LLMs. In Liakata, M.; Mor- eira, V. P.; Zhang, J.; and Jurgens, D., eds., Proceedings of the 64th Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, 122–146. Association for Computational Linguistics. Xu, B.; Xue, Z.; Chen, D.; Fu, C.; Wu, C.; Huang, C.; Jiang, C.; Fang, J.; Deng, X.; Chen, Y.; Yao, Y.; Wang, X.; Shang, J.; Yu, G.; and Zhang, N. 2026b. TokenPilot: Cache- Efficient Context Management for LLM Agents. CoRR, abs/2606.17016. Xu, W.; Li, S.; Ye, T.; Cao, Q.; Chen, Y.; et al. 2026c. ResearchClawBench: A Benchmark for End-to-End Au- tonomous Scientific Research. CoRR, abs/2606.07591. Yamada, Y.; Lange, R. T.; Lu, C.; Hu, S.; Lu, C.; et al. 2025. The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. CoRR, abs/2504.08066. Ye, H.; Lin, H.; Tang, J.; Luo, Y.; Yang, C.; Su, C.; Thapa, R.; Yang, R.; Liu, R.; Li, Z.; Gao, C.; Ding, D.; He, G.; Zhang, M.; Sun, L.; Wang, W.; Zhong, Y.; Shen, Z.; He, D.; Ma, J.; Ermon, S.; Li, T.; Chu, X.; Wang, J. Z.; and Xu, Y. 2026. Evaluation-driven Scaling for Scientific Discovery. CoRR, abs/2604.19341. Yehudai, A.; Eden, L.; Li, A.; Uziel, G.; Zhao, Y.; Bar- Haim, R.; Cohan, A.; and Shmueli-Scheuer, M. 2026. A Survey on Evaluation of LLM-based Agents. In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds., Findings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2-7, 2026, 26690– 26714. Association for Computational Linguistics. Yu, J.; Xu, Z.; Wang, J.; and Yang, Y. 2026. Think-with- Rubrics: From External Evaluator to Internal Reasoning Guidance. CoRR, abs/2605.07461. Z.ai. 2026. GLM-5.2: Built for Long-Horizon Tasks. https: //z.ai/blog/glm-5.2. Accessed: 2026-07-24. Zheng, J.; Zhang, J.; Luo, Y.; Mao, Y.; Gao, Y.; Du, L.; Chen, H.; and Zhang, N. 2026. Can We Predict Before Exe- cuting Machine Learning Agents? In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds., Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, 3941–3974. Asso- ciation for Computational Linguistics. Zhu, M.; Wei, C.; Xu, J.; Cheng, Y.; Chen, Z.; and He, J. 2026. DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents. CoRR, abs/2606.17029. A Additional Experimental and Methodological Details A.1 Benchmark and Evaluation Details ResearchClawBench. ResearchClawBench contains 40 end-to-end scientific research tasks, with four tasks from each of ten domains: astronomy, chemistry, earth science, energy science, information science, life science, materials science, mathematics, neuroscience, and physics. Each task is con- structed from a published target paper with a clearly defined scientific question and accessible raw data. The evaluated agent is provided with a task instruction, related literature, raw data, and a workspace in which it can inspect files, im- plement analyses, execute experiments, generate figures, and write a final scientific report. The hidden target paper and its official evaluation rubric are not accessible to the agent during rubric induction, research execution, verification, or revision. We use the official ResearchClawBench evaluator with- out modifying its scoring protocol. Each benchmark rubric consists of weighted criteria associated with concrete scien- tific artifacts in the hidden target paper. The rubric contains both textual criteria, which assess scientific claims, meth- ods, quantitative results, and explanations, and image crite- ria, which assess figure-level evidence. GPT-5.1 is used to evaluate the final report and its supporting artifacts against these criteria. The resulting task score ranges from 0 to 100, with a score around 50 intended to represent approximately target-paper-level re-discovery under the benchmark proto- col. For each configuration, the overall score is the arithmetic mean over all 40 tasks. Each domain score is the arithmetic mean over the four tasks belonging to that domain. All re- ported improvements are absolute differences between paired vanilla and +AutoSciRub scores on the same task set. AstaBench End-to-End Discovery. We additionally eval- uate AutoSciRub on a fixed subset of 20 tasks from the Easy split of the AstaBench End-to-End Discovery benchmark. The subset is sampled once from the original 40-task test set using random seed 20260707. The sampling procedure does not preserve the order of the source dataset; instead, the sampled order is retained as the execution order. The same fixed subset and execution order are used for all evaluated configurations. Table 4 lists the exact task identifiers and task names. Each AstaBench task requires an agent to complete an end-to-end research cycle involving experimental implemen- tation, execution, result analysis, and technical report gener- ation. We use the official task-specific rubrics and LLM-as- judge evaluation pipeline, with MiniMax-M3 serving as the evaluator. For every rubric criterion, the evaluator independently ex- amines three output facets: the generated report, the gen- erated code, and the supporting artifacts, including datasets, execution logs, figures, and model outputs. Each facet is clas- sified as meets criterion, fails criterion, or no evidence either way. The facet-level assessments are then combined into a bi- nary verdict for the corresponding criterion. The official task score is the arithmetic mean of the binary criterion verdicts, multiplied by 100 for presentation. In addition to the mean evaluation score, we report the number of successfully completed tasks. A task is consid- ered successfully completed when the agent run terminates without an execution-level failure and produces a valid sub- mission that can be processed and scored by the official evaluator. Models and agent harnesses. We evaluate generalization along two complementary axes on ResearchClawBench. In cross-model evaluation, we hold the Codex agent harness fixed and vary the backbone LLM among GPT-5.4, GLM- 5.2, and MiniMax-M3. In cross-harness evaluation, we hold the DeepSeek-V4-Flash backbone fixed and vary the sur- rounding agent implementation among Claude Code, Open- Claw, and OpenScience. For AstaBench, we evaluate three representative configu- rations: Claude Code with DeepSeek-V4-Flash, OpenClaw with DeepSeek-V4-Flash, and Codex with GPT-5.4-mini. The complete configurations and corresponding benchmark evaluators are summarized in Table 5. For every paired comparison, the vanilla setting denotes the original agent without AutoSciRub. The +AutoSciRub setting uses the same backbone model, agent harness, task instruction, task-visible data, workspace, and available tools, while additionally performing rubric induction, criterion- Order Task IDTask name 1 idea-226-simplifiedsimple-knowledge-enhanced-vqa 2 idea-206-simplifiedselective-entity-quantization 3 idea-131-simplifiedsimple-semantic-backdoor-defense 4 idea-21-simplifiedtext-to-code-prompt-retrieval 5 idea-78-simplifiedsimple-bias-detection-worldmodel 6 idea-337-simplifiedmedical-verbalization-calibration 7 idea-17-simplifiedtopic-focused-memory-summarization 8 idea-86-simplifiedsimple-language-skill-transfer 9 idea-259-simplifiedsimple-relevance-metrics 10 idea-174-simplifiedsimple-position-aware-prompts 11 idea-5-simplifiedsimple-entity-debate 12 idea-159-simplifiedconfidence-based-contamination-detection 13 idea-304-simplifieddual-task-adversarial-robustness 14 idea-192-simplifiedsimple-fact-unlearning 15 idea-147-simplifiedmini-socratic-code-generation 16 idea-301-simplifiedsimplified-task-learning-transfer 17 idea-134-simplifiedsimple-cross-lingual-backdoor 18 idea-278-simplifiedvisual-entity-knowledge-test 19 idea-68-simplifiedevolutionary-prompt-optimization 20 idea-141-simplifiedbiobert-pet-ner Table 4: The fixed 20-task subset used for AstaBench evaluation, listed in the retained execution order. The tasks were sampled from the original 40-task test set using random seed 20260707. BenchmarkSettingAgent harnessBackbone LLMEvaluator ResearchClawBenchCross-modelCodexGPT-5.4GPT-5.1 ResearchClawBenchCross-modelCodexGLM-5.2GPT-5.1 ResearchClawBenchCross-modelCodexMiniMax-M3GPT-5.1 ResearchClawBenchCross-harnessClaude CodeDeepSeek-V4-FlashGPT-5.1 ResearchClawBenchCross-harnessOpenClawDeepSeek-V4-FlashGPT-5.1 ResearchClawBenchCross-harnessOpenScienceDeepSeek-V4-FlashGPT-5.1 AstaBenchTransferClaude CodeDeepSeek-V4-FlashMiniMax-M3 AstaBenchTransferOpenClawDeepSeek-V4-FlashMiniMax-M3 AstaBenchTransferCodexGPT-5.4-miniMiniMax-M3 Table 5: Model, agent-harness, and evaluator configurations used in the ResearchClawBench and AstaBench experiments. level verification, and targeted revision. Accordingly, the paired configurations are controlled for the underlying agent and execution environment, but are not compute-matched, because AutoSciRub introduces the additional model calls required by the method. Execution and evaluation protocol. Each task– configuration pair is executed once, producing one scientific report and one set of supporting artifacts. Each resulting submission is then independently evaluated three times using the same benchmark evaluator and evaluation protocol. The arithmetic mean of the three judge scores is used as the task-level score. For ResearchClawBench, the overall score is computed as the arithmetic mean over all 40 task-level scores, and each domain score is computed over its four constituent tasks. For AstaBench, the mean score is computed over the fixed 20-task subset. Successful-task counts are computed independently from the rubric-based evaluation scores. Within each paired comparison, the vanilla and +AutoSciRub settings receive the same task inputs, task- visible resources, workspace, and tool access. Available tools include the native capabilities of the corresponding agent har- ness, such as file inspection, code execution, and web search, as well as the external retrieval services used by AutoSciRub. The hidden target paper and the official benchmark evaluation rubric remain inaccessible to both settings during research execution. We do not impose a shared explicit cap on tokens, tool calls, wall-clock runtime, or monetary cost across heteroge- neous agent harnesses. Each system instead operates under the execution, context, and tool constraints of its native envi- ronment. Our primary evaluation therefore focuses on output quality rather than budget-normalized efficiency. Rubric-guided verification and revision use a maximum of three revision rounds with adaptive early stopping. To dis- tinguish the effect of structured rubric guidance from that of additional opportunities for revision, we additionally evalu- ate a three-round rubric-free self-refinement baseline starting from the same initial reports and using the same underlying research agent. The complete verification, revision, early- stopping, and self-refinement protocols are described in the following subsection. A.2 Rubric Quality Evaluation Compared rubric variants and scoring protocol. For each of the 40 ResearchClawBench tasks, we evaluate two rubric variants: the instruction-derived rubric skele- ton produced after Rubric Skeleton Induction, and the task- specific executable rubric produced after Scientific Litera- ture Grounding, Task-Data Exploration, and Criterion Syn- thesis. Each rubric is evaluated along four dimensions: Speci- ficity, Actionability, Evidence Verifiability, and Scientific Core Coverage. All dimensions use a five-point ordinal scale, where a higher value indicates stronger rubric quality. We use MiniMax-M3 as the rubric-quality judge for all evaluations. To reduce cross-dimensional interference, we use four stan- dalone judge prompts, each of which evaluates exactly one dimension. The judge receives only one candidate rubric in each call and is explicitly instructed not to score the other three dimensions. For every judge call, the evaluator is provided with the task instruction, a description of the task-visible data, the candidate rubric, and the official benchmark checklist. The checklist is used only as evaluator-side evidence for identi- fying the intended scientific success conditions. It is never exposed to the agent during rubric construction, research ex- ecution, verification, or revision. The judge is instructed to assess semantic and scientific equivalence rather than lexical overlap with the checklist. The rubric skeleton and executable rubric are evaluated in separate calls. The judge is not provided with the generated report or the alternative rubric variant. The overall quality score of rubricR is computed as the unweighted arithmetic mean of the four dimension scores: Q(R) = 1 4 (s spec + s act + s ver + s core ), where each score is an integer between 1 and 5. Evaluation dimensions. Specificity. Specificity measures how precisely the rubric states what is to be done and pro- duced. A specific rubric names the relevant methods or exper- iments, variables, conditions, metrics, comparators, thresh- olds, and expected artifacts at a level that makes the intended analysis unambiguous. Specificity concerns the concreteness of the specification; it does not by itself establish that the plan is executable, scientifically valid, or directed at the correct scientific objective. Actionability. Actionability measures whether a compe- tent agent can execute the rubric using the available inputs without inventing consequential parts of the plan. An ac- tionable rubric provides sufficient operational steps, depen- dencies, parameter or method choices, decision rules, and expected outputs to support implementation. Actionability does not reward detail that is irrelevant to execution and does not guarantee that an executable plan addresses the task’s scientific core. Evidence Verifiability. Evidence Verifiability measures whether the rubric requires inspectable and traceable ev- idence that permits independent verification of both task completion and the resulting scientific claims. Verifiable evidence may include quantitative tables, figures, machine- readable outputs, controls, baseline comparisons, uncertainty estimates, provenance, and explicit links between claims and generated results. This dimension evaluates the checkability and diagnostic value of the evidence rather than the number of artifacts requested. Scientific Core Coverage. Scientific Core Coverage mea- sures the extent to which the rubric faithfully covers and op- erationalizes the task’s core scientific success conditions. It evaluates whether the rubric’s objectives, methods or scien- tifically equivalent alternatives, metrics, comparisons, arti- facts, scope, conventions, evidence, and required conclusions collectively target the intended scientific problem. Scores decrease when important success conditions are omitted or underweighted, when the rubric drifts toward an adjacent problem, or when core goals are replaced by non-equivalent proxies. Alternative methods and artifacts receive full credit when they provide scientifically equivalent or stronger evi- dence for the same core claim. The four dimensions capture complementary properties. A rubric may be highly specific and actionable but receive a low Scientific Core Coverage score if it precisely opera- tionalizes the wrong analysis. Conversely, a concise rubric may identify the correct scientific core but receive lower Specificity or Actionability scores if important implemen- tation choices remain unspecified. Evidence Verifiability is assessed separately: a scientifically aligned and executable plan may still be difficult to verify when it does not require inspectable outputs that directly support its conclusions. Complete rubric-quality scoring prompts. The follow- ing four prompts are used independently. In the JSON tem- plates, the value of score is replaced with an integer from 1 to 5. Rubric item identifiers refer to goal IDs for rubric skeletons and criterion IDs for executable rubrics. Complete Rubric-Quality Scoring Prompts The following four prompts independently evaluate Specificity, Actionability, Evidence Verifiability, and Scientific Core Coverage. Each judge call receives one candidate rubric and evaluates exactly one dimension. Specificity Scoring Prompt You are a senior scientific−methods reviewer. Evaluate exactly one construct: SPECIFICITY. Definition: Specificity is the precision with which the rubric states what is to be done and produced. It concerns whether the relevant methods or experiments, variables, conditions, metrics, comparators, thresholds, and expected artifacts are concrete enough to make the intended analysis unambiguous. Evaluation rules: 1. Inspect the rubric’s core items rather than rewarding length or technical jargon. 2. Credit concrete methods, conditions, variables, metrics, comparisons, thresholds, named outputs, and artifact requirements when they remove genuine ambiguity. 3. Do not reward irrelevant detail or copied wording from the task or benchmark checklist. 4. Do not lower the score merely because the specified plan may be difficult to execute, scientifically questionable, weakly evidenced, or misdirected; those properties belong to other dimensions. 5. Cite rubric item IDs and identify the most consequential specified and unspecified details. Score anchors: 1 = The rubric is almost entirely generic; core methods, conditions, metrics, and outputs are not concretely specified. 2 = A few concrete details are present, but most consequential aspects remain ambiguous. 3 = The main analysis is identifiable, but important conditions, metrics, comparisons, thresholds, or artifacts are underspecified. 4 = Nearly all core details are explicit; only minor ambiguities remain. 5 = The core analysis is precise and unambiguous across methods, conditions, variables, metrics, comparisons, thresholds, and expected artifacts. Return JSON only. Set "score" to an integer from 1 to 5: "dimension": "specificity", "score": 1, "evidence": "Concise justification citing rubric item IDs.", "specified_strengths": ["..."], "remaining_ambiguities": ["..."] TASK INSTRUCTION: [task_instruction] AVAILABLE−DATA DESCRIPTION: [available_data] BENCHMARK CHECKLIST− EVALUATOR−ONLY CONTEXT: [benchmark_checklist] RUBRIC TO SCORE: [rubric] Actionability Scoring Prompt You are a senior scientific−methods reviewer. Evaluate exactly one construct: ACTIONABILITY. Definition: Actionability is the degree to which a competent agent can execute the rubric using the available inputs without inventing consequential parts of the plan. It concerns operational steps, dependencies, method and parameter choices, decision rules, and deliverables that reduce implementation uncertainty. Evaluation rules: 1. Determine whether the rubric supplies an executable route from the available data to the required outputs. 2. Identify consequential methods, parameters, dependencies, preprocessing choices, decision rules, or deliverables that the agent would otherwise have to invent. 3. Credit flexibility when multiple choices are genuinely interchangeable and the rubric supplies a sound decision rule or acceptable range. 4. Do not reward detail that does not help execution. Do not judge whether the chosen scientific target is the correct one; that belongs to Scientific Core Coverage. 5. Cite rubric item IDs and focus on implementation consequences. Score anchors: 1 = No executable route is supplied; the agent must design nearly the entire plan. 2 = Major methods, dependencies, parameters, or decisions must be invented. 3 = An executable outline exists, but some consequential implementation choices remain unresolved. 4 = The core workflow is directly executable, with only minor or low−impact choices left open. 5 = An end−to−end operational plan specifies all consequential steps, dependencies, decision rules, and outputs needed for execution. Return JSON only. Set "score" to an integer from 1 to 5: "dimension": "actionability", "score": 1, "evidence": "Concise justification citing rubric item IDs.", "executable_elements": ["..."], "missing_decisions": ["..."] TASK INSTRUCTION: [task_instruction] AVAILABLE−DATA DESCRIPTION: [available_data] BENCHMARK CHECKLIST− EVALUATOR−ONLY CONTEXT: [benchmark_checklist] RUBRIC TO SCORE: [rubric] Evidence Verifiability Scoring Prompt You are a senior scientific−methods reviewer. Evaluate exactly one construct: EVIDENCE VERIFIABILITY. Definition: Evidence Verifiability is the degree to which the rubric requires inspectable and traceable evidence that allows an independent reviewer to verify task completion and evaluate whether the scientific claims follow from the generated results. Evaluation rules: 1. Inspect required quantitative tables, figures, machine−readable outputs, controls, baseline comparisons, uncertainty estimates, provenance, and claim−to−evidence links when scientifically applicable. 2. Judge the diagnostic and confirmatory value of evidence, not the number of files. 3. Credit explicit must−show criteria, metrics, uncertainty, comparison targets, and traceable paths that make an artifact independently checkable. 4. Penalize rubrics that allow conclusions to be asserted without supporting results, or request decorative outputs that cannot test the claims. 5. Do not judge whether the rubric targets the correct scientific goal; that belongs to Scientific Core Coverage. Score anchors: 1 = Conclusions can be asserted without any inspectable supporting evidence. 2 = Outputs are requested only vaguely or lack checkable criteria and claim links. 3 = Some core results are inspectable, but important claims, comparisons, uncertainty, or diagnostics remain weakly supported. 4 = Core claims are supported by checkable quantitative artifacts and appropriate diagnostics; only minor traceability gaps remain. 5 = The evidence chain is complete and traceable from method to result to claim, including controls, uncertainty, comparisons, and provenance where relevant. Return JSON only. Set "score" to an integer from 1 to 5: "dimension": "evidence_verifiability", "score": 1, "evidence": "Concise justification citing rubric item IDs.", "verifiable_artifacts": ["..."], "verification_gaps": ["..."] TASK INSTRUCTION: [task_instruction] AVAILABLE−DATA DESCRIPTION: [available_data] BENCHMARK CHECKLIST− EVALUATOR−ONLY CONTEXT: [benchmark_checklist] RUBRIC TO SCORE: [rubric] Scientific Core Coverage Scoring Prompt You are a senior scientific−methods reviewer. Evaluate exactly one construct: SCIENTIFIC CORE COVERAGE. Definition: Scientific Core Coverage is the extent to which the rubric faithfully covers and operationalizes the task’s core scientific success conditions through aligned objectives, scientifically equivalent methods, metrics, comparisons, artifacts, scope, conventions, evidence, and required conclusions. It excludes material omission, goal drift, and substitution by adjacent but non−equivalent proxies. Evaluation rules: 1. Infer the task’s core scientific success conditions from the instruction and available data. Use the benchmark checklist and its criterion weights, when available, only as evaluator−side evidence of importance. 2. Map each core condition to rubric items and classify it as aligned, scientifically equivalent, partial, proxy, missing, or conflicting. 3. Judge semantic and scientific−function equivalence, never wording overlap. Alternative methods and artifacts deserve full credit when they provide equivalent or stronger evidence for the same core claim. 4. Penalize omission or underweighting of high−importance goals, goal drift, non−equivalent proxy substitution, artifact or metric mismatch, scope dilution, incompatible conventions, and conclusion anchoring. 5. Missing low−level implementation detail alone belongs to Specificity or Actionability and must not reduce this score when the scientific target remains intact. Score anchors: 1 = The rubric operationalizes a substantially different problem, contradicts a core success condition, or is dominated by non−equivalent proxies. 2 = Major omissions, conflicts, or proxy substitutions affect one or more core or high−importance success conditions. 3 = The main objective is recognizable, but an important success condition is missing, weakened, underweighted, or replaced. 4 = Core success conditions are faithfully operationalized; only minor or peripheral mismatches remain, and valid equivalent alternatives are used appropriately. 5 = All core success conditions are tightly covered by aligned or scientifically equivalent objectives, methods, metrics, comparisons, artifacts, scope, and evidence, with no material drift, omission, or proxy replacement. Return JSON only. Set "score" to an integer from 1 to 5. The allowed values of "importance" are "core", "important", and "peripheral". The allowed values of "status" are "aligned", "equivalent", "partial", "proxy", "missing", and "conflicting": "dimension": "scientific_core_coverage", "score": 1, "evidence": "Concise justification citing rubric item IDs and core consequences.", "criterion_audit": [ "criterion": "...", "importance": "core", "status": "aligned", "rubric_items": ["R1"], "rationale": "..." ], "core_gaps_or_drift": ["..."] TASK INSTRUCTION: [task_instruction] AVAILABLE−DATA DESCRIPTION: [available_data] BENCHMARK CHECKLIST− EVALUATOR−ONLY CONTEXT: [benchmark_checklist] RUBRIC TO SCORE: [rubric] A.3 Rubric-Guided Revision Evaluation Experimental configuration and shared initialization. We compare rubric-guided revision with rubric-free holis- tic self-refinement on all 40 ResearchClawBench tasks. All checkpoint-0 reports, verification calls, targeted revisions, and rubric-free revisions are executed with OpenClaw us- ing deepseek/deepseek-v4-flash. There is no sep- arately instantiated verifier model. Instead, the same Open- Claw agent invokes the report-verifier skill within the task session. Both revision strategies start from the same set of 40 frozen checkpoint-0 reports and figures. These checkpoint-0 outputs are generated under the task-specific executable rubric after planning, literature grounding, experiment design, experi- ment execution, figure generation, and initial report writing, but before any verifier call or verifier-driven revision. Their mean ResearchClawBench score is 18.309083, reported as 18.31 in the main paper. For the rubric-free condition, the canonical checkpoint- 0 report.md file and its figures are copied into a new isolated workspace. SHA-256 hashes are computed before and after copying to verify that the starting report and fig- ures are unchanged. We do not claim that the two strategies share an identical complete workspace: checkpoint-0 code and output directories are not frozen or copied because they may contain rubric-related state and later feedback. Instead, the rubric-free workspace is supplied again with the original instruction, task data, and related work, and the agent may create new code, outputs, and analyses during refinement. Criterion-level verification. Before each guided revision, the current artifact is evaluated by the same OpenClaw agent through a verification-only call. The verifier examines the current report, generated experiments and results, figures and tables, quantitative values, conclusions, and mechanistic or analytical explanations. It also checks whether the artifacts required by the task-specific executable rubric are present or explicitly represented in the report, including their filenames, titles, captions, display names, and detailed descriptions. The verifier is instructed to distinguish generated exper- imental evidence from statements supported only by prior literature. Conclusions are considered adequately supported only when they follow from evidence produced within the current task execution. For each scientific goal, the verifier records whether the report contains the required experiment, results, figure or table, conclusion, and mechanism or analytical explanation. It also evaluates individual rubric items and produces sta- ble feedback items specifying the remaining issue and the required corrective action. The top-level overall_pass field is an LLM judgment rather than a deterministic conjunction of all criterion-level Boolean values. An artifact passes when all scientific goals have sufficient experimental support, interpretation, and con- clusion coverage, and all high-priority rubric items are ei- ther satisfied or explicitly and scientifically justified by the generated evidence. Lower-priority items are not required to pass individually. The verifier does not produce a sep- arate structured not_applicable state; an inapplicable requirement may instead be justified in free text while the corresponding structured field remains Boolean. The expected verifier output follows the schema below. Verifier Output Schema "source": "rcbench_report_verifier", "overall_pass": false, "goal_checks": [ "goal_id": "G1", "pass": false, "coverage": "experiment": true, "results": true, "figure_or_table": false, "conclusion": true, "mechanism_or_analysis": false , "missing": ["..."], "fixes": ["..."] ], "rubric_item_checks": [ "rubric_item_id": "R1", "pass": false, "missing": ["..."], "fixes": ["..."] ], "feedback_items": [ "feedback_id": "F001", "goal_id": "G1", "rubric_item_id": "R1", "issue": "...", "required_action": "..." ], "artifact_checks": [ "path/name, present?, described?, issue" ], "required_iterations": ["..."] The runner reads the top-level overall_pass value directly and does not independently reconstruct the pass de- cision from the individual checks. If the verifier output is missing or cannot be parsed, the runner treats the artifact as not passing and proceeds to revision. No separate JSON- schema validation or verifier-specific retry is performed. A task run is marked as failed only when the OpenClaw process exits with a nonzero status. Targeted revision and early stopping. If verification does not pass, the current report_rubric_check.json is supplied to the same OpenClaw agent for one targeted revi- sion. The revision call has access to the current workspace, INSTRUCTIONS.md, the current report, the executable rubric, goal-level experiment plans, existing code, outputs, figures, and the current verifier feedback. The agent may modify code, rerun or add experiments and analyses, regenerate figures or tables, correct numerical results, revise unsupported claims, and update the complete report/report.md. The revision call is instructed not to invoke the verifier itself; verification and revision are ex- ecuted as separate agent calls. We allow at most three revision rounds. The resulting execution sequence is V (0) → R (1) → V (1) → R (2) → V (2) → R (3) → V (3) , where V (t) denotes verification of checkpoint t, and R (t) denotes the revision producing checkpoint t. Consequently, each task receives at most three revision calls and four ver- ification calls. The procedure terminates immediately when a verification call returns overall_pass=true. If the artifact does not pass earlier, a final verification is performed after the third revision. Adaptive stopping and checkpoint aggregation. Rubric- guided revision generates and evaluates only reports that are actually produced. Across the 40 tasks, the experiment contains 103 distinct report checkpoints: 40 checkpoint-0 reports, 40 checkpoint-1 reports, 17 checkpoint-2 reports, and 6 checkpoint-3 reports. When a task passes verification, no artificial reports are generated for later revision budgets. Instead, its most re- cent real report and its previously obtained score are carried forward when computing later checkpoint-level means. The carried-forward report is not rerun or rescored. Thus, every reported guided checkpoint mean is computed over all 40 tasks, while only tasks that have not yet stopped produce a new report at the next revision round. Each actually generated report checkpoint is indepen- dently scored three times by the GPT-5.1 ResearchClaw- Bench judge, and the arithmetic mean of the three scores is used as its task-level score. Rubric-free holistic self-refinement. The rubric-free baseline starts from the same frozen checkpoint-0 re- port and figures and uses the same OpenClaw and deepseek/deepseek-v4-flash configuration. It is explicitly prohibited from accessing, constructing, inferring, or searching for the task-specific executable rubric, the hid- den benchmark checklist, the target study, rubric scores, or prior verifier feedback. At each round, the agent invokes the holistic-self-review skill and performs a holistic scientific review followed immediately by one revision within the same agent call. The agent may use tools, write code, rerun or add experiments and analyses, modify figures and tables, revise results and claims, and update the complete report. Unlike rubric-guided revision, rubric-free self-refinement does not use verifier-based early stopping. All 40 tasks exe- cute three review–revision cycles, producing 40 real reports at each of checkpoints 1, 2, and 3. In the first round, the runner requires between one and six prioritized issues. In the second and third rounds, an empty issue list and a no-op revi- sion are allowed when no substantive deficiency is identified. Each resulting report is independently scored three times by the same GPT-5.1 judge. This baseline controls for repeated opportunities to inspect and revise the report, but it is not a strictly compute-matched or workspace-identical control. Guided revision separates verification and revision into two calls and uses adaptive stopping, whereas rubric-free refinement combines review and revision into one call and always performs three rounds. Guided revision also retains the original code and output state, while the rubric-free condition starts from the frozen report and figures in a newly initialized workspace. B Rubric Output Formats This section presents the output formats of the instruction-derived rubric skeleton and the task-specific executable rubric. One complete entry is shown for each format; the remaining entries are omitted for brevity. Instruction-derived rubric skeleton. The rubric skeleton represents the task as a set of atomic scientific goals. Each goal contains a unique identifier, a concise title, and a concrete task description. Instruction-Derived Rubric Skeleton "goals": [ "goal_id": "G1", "title": "Characterize spatial concentration of cloud−seeding projects", "task": "Analyze the geographic distribution of cloud−seeding projects across U.S. states using the published records, and identify which states account for the majority of reported activity." , "...": "Additional scientific goals are omitted." ] Task-specific executable rubric. The executable rubric augments the scientific goals with concrete experiments, required evidence artifacts, metrics, comparisons, data sources, expected conclusions, priorities, and execution constraints. Task-Specific Executable Rubric "source": "rcbench_experiments_design", "task_summary": "Reproduce and validate the central empirical conclusions from a published NOAA weather−modification dataset of U.S. cloud seeding projects (2000−2025). Generate spatial concentration maps, annual activity dynamics, purpose composition, and agent−apparatus deployment patterns using transparent, script−based analysis.", "selected_environment": ".venv−rcb−base", "environment_reason": "The selected environment provides the packages required for tabular analysis, statistical computation, and geospatial visualization.", "selected_strategy": "Independently compute the required statistics and figures from the structured dataset and synthesize the results into an empirical assessment.", "rubric_items": [ "id": "R1", "linked_goal_ids": [ "G1" ], "goal": "Quantify state−level spatial concentration of cloud seeding projects and produce a choropleth map showing geographic distribution.", "experiment_plan_path": "outputs/goals/G1/experiment_plan.md", "experiments_to_run": [ "Group records by state and count projects", "Compute rank, percentage share, and Gini coefficient", "Merge state counts with GeoJSON and generate a choropleth map", "Create a state−by−purpose heatmap" ], "required_artifacts": [ "type": "figure", "name": "spatial_distribution.png", "expected_path": "report/images/spatial_distribution.png", "must_show": "A choropleth map of U.S. states colored by the number of cloud seeding projects, showing strong western concentration and identifying California, Colorado, Texas, and Utah among the leading states.", "description_requirement": "Explain the geographic pattern of cloud seeding activity and the factors associated with western concentration." , "type": "figure", "name": "state_purpose_heatmap.png", "expected_path": "report/images/state_purpose_heatmap.png", "must_show": "A state−by−purpose heatmap demonstrating how project purposes vary across states.", "description_requirement": "Describe how project purposes vary across states and regions." , "type": "table", "name": "top_states_table", "expected_path": "report/report.md", "must_show": "A ranked list of the leading states with project counts and percentage shares.", "description_requirement": "Summarize the state−level concentration statistics." ], "metrics_or_values": [ "Projects per state", "Percentage share of the top three and top five states", "Gini coefficient" ], "baselines_or_comparisons": [ "Expected geographical concentration in water−stressed western states" ], "data_sources": [ "data/dataset1_cloud_seeding_records/cloud_seeding_us_2000_2025.csv[state]", "data/dataset1_cloud_seeding_records/us_states.geojson" ], "expected_conclusions": [ "Cloud seeding is heavily concentrated in western U.S. states", "California has the highest project count", "The spatial distribution reflects water−resource management priorities" ], "priority": "high", "risk_if_missing": "Loss of the primary spatial evidence supporting geographic concentration claims." , "...": "Additional executable rubric items are omitted." ], "execution_plan": [ "...": "Task−level execution steps are omitted." ], "claims_to_avoid": [ "Do not claim causal relationships between cloud seeding and precipitation changes.", "... additional unsupported claims are omitted ..." ]