Paper deep dive
RSI-Forge: From Research Papers to Environments for Recursive Self-Improvement
Renxiong Wang, Darvin Yi, Abril Herrlein, Anas Mahmoud, Advait Gosai, Lisiman Hua, MohammadHossein Rezaei, Xingang Guo, Anisha Gunjal, Utkarsh Tyagi, David J. Lee, Minglai Yang, Haris Riaz, Chenguang Wang, Huaxiu Yao, Daniel Yue Zhang, Aakash Sabharwal, Tong Zhao, Yunzhong He
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 86%
Last extracted: 10/8/2026, 7:43:40 AM
Summary
The paper introduces RSI-Forge, a multi-agent pipeline that automatically converts published research papers into executable environments for recursive self-improvement (RSI). The system uses three agents (author, reproducer, reviewer) to construct tasks, establish baselines via independent reproduction, and verify quality. The authors generated 210 environments across 18 fields, demonstrating that models can improve upon reproduced baselines in 68 out of 120 tested environments, with most improvements involving work beyond simple parameter tuning.
Entities (26)
Relation Signals (27)
RSI-Forge → generates → 210 environments
confidence 98% · We present 210 environments across 18 fields
RSI-Forge → basedon → published papers
confidence 95% · RSI-Forge... turns published papers into executable environments
RSI-Forge → relatedto → Recursive self-improvement
confidence 95% · Environments are the foundation of recursive self-improvement
RSI-Forge → uses → Author Agent
confidence 95% · An author agent develops the task, starting solution, and evaluator.
RSI-Forge → uses → Reviewer Agent
confidence 95% · A reviewer agent checks that the task can be understood from its instructions...
RSI-Forge → uses → Reproducer Agent
confidence 95% · A reproducer agent independently implements the paper’s method... establishing the reproduced baseline.
RSI-Forge → outperforms → reproduced paper methods
confidence 92% · Models also outperform the reproduced paper methods in 68 of the 120 environments
Reproducer Agent → establishes → reproduced baseline
confidence 90% · The reproducer agent independently implements the paper’s method... to establish the reproduced baseline score.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Environments are the foundation of recursive self-improvement: they provide the problems agents work on and the feedback used to evaluate progress. Yet constructing challenging research environments with reliable evaluation still depends on domain experts, limiting their scale and disciplinary coverage. We introduce RSI-Forge, a multi-agent pipeline that turns published papers into executable environments for self-improvement. Three agents coordinate construction, reproduction, and review to produce tasks with automated evaluators; each paper's method is independently reimplemented to establish a baseline score. We present 210 environments across 18 fields, including 90 reviewed by independent human domain experts. Both experts and agent judges give high ratings to the potential for improving the provided starting solutions and the evaluators' ability to distinguish solution quality, whereas experts are more critical of shortcut resistance, faithfulness to the source paper, and whether a single idea can exhaust a task. To validate their use for repeated improvement, we evaluate four models over 3 successive attempts on 120 environments, with each attempt inheriting prior code and notes while model weights remain fixed. At least one model improves after the first attempt in 84% of environments. Models also outperform the reproduced paper methods in 68 of the 120 environments, demonstrating room for gains beyond these baselines. Transcript analysis identifies work beyond parameter tuning in 95% of these successful attempts. Analysis of the resulting trajectories shows that models scoring lower on these tasks explore less, more often accept gains smaller than the reported standard error, and rely more heavily on tuning to the development set. RSI-Forge provides a scalable approach to constructing research environments for training and evaluating self-improving agents.
Tags
Links
- Source: https://arxiv.org/abs/2610.09426v1
- Canonical: https://arxiv.org/abs/2610.09426v1
Trouble viewing inline? Open PDF directly →
Full Text
69,557 characters extracted from source content.
Expand or collapse full text
RSI-Forge: From Research Papers to Environments for Recursive Self-Improvement Renxiong Wang 1 , Darvin Yi 1 , Abril Herrlein 1 , Anas Mahmoud 1 , Advait Gosai 1 , Lisiman Hua 1 , MohammadHossein Rezaei 1 , Xingang Guo 1 , Anisha Gunjal 1 , Utkarsh Tyagi 1 , David J. Lee 1 , Minglai Yang 1 , Haris Riaz 1 , Chenguang Wang 1,2 , Huaxiu Yao 3 , Daniel Yue Zhang 1 , Aakash Sabharwal 1 , Tong Zhao 1 , Yunzhong He 1 1 Scale AI, 2 University of California, Santa Cruz, 3 University of North Carolina at Chapel Hill renxiong.wang,darvin.yi,tong.zhao,yunzhong.he@scale.com Abstract Environments are the foundation of recursive self-improvement: they provide the problems agents work on and the feedback used to evaluate progress. Yet constructing challenging research en- vironments with reliable evaluation still depends on domain experts, limiting their scale and disciplinary coverage. We introduce RSI-Forge, a multi-agent pipeline that turns published pa- pers into executable environments for self-improvement. Three agents coordinate construction, reproduction, and review to produce tasks with automated evaluators; each paper’s method is independently reimplemented to establish a baseline score. We present 210 environments across 18 fields, including 90 reviewed by independent human domain experts. Both experts and agent judges give high ratings to the potential for improving the provided starting solutions and the evaluators’ ability to distinguish solution quality, whereas experts are more critical of shortcut resistance, faithfulness to the source paper, and whether a single idea can exhaust a task. To validate their use for repeated improvement, we evaluate four models over 3 successive attempts on 120 environments, with each attempt inheriting prior code and notes while model weights remain fixed. At least one model improves after the first attempt in 84% of environments. Models also outperform the reproduced paper methods in 68 of the 120 environments, demonstrating room for gains beyond these baselines. Transcript analysis identifies work beyond parameter tuning in 95% of these successful attempts. Analysis of the resulting trajectories shows that models scoring lower on these tasks explore less, more often accept gains smaller than the reported standard error, and rely more heavily on tuning to the development set. RSI-Forge provides a scalable approach to constructing research environments for training and evaluating self-improving agents. 1 Introduction Recursive self-improvement (RSI) is a process in which an agent improves its own capabilities and uses those improvements to guide further changes [5]. Such agents could accumulate useful strategies and 1 arXiv:2610.09426v1 [cs.AI] 7 Oct 2026 Scale AI Research Filter by criteria 10,800 preprints 2,117 eligible — 20% Build, review, reproduce, review authorreviewreproducerreview returned for repair rejected accepted reviewed and repaired Delivered 210 environments 18 fields Tested with models Opus 5 0.80 GPT-5.6-sol 0.76 DeepSeek 0.65 Haiku 4.5 0.33 author reproducer reviewer normalized score withheld inputs none during environment construction receivesrole paper and construction requirementsconstructs the task and evaluator paper and solver-facing materialsestablishes the reproduced baseline downstream evaluation resultsapproves the environment or requests repairs agent environment and reproduction report author agent's method implementation Figure 1. Overview of RSI-Forge and the information available to each agent. Failed checks lead to repair or rejection before an environment is accepted. The output count includes all 210 environments. The performance bars report median scores of final solutions after 3 successive attempts on 120 environments sampled from the main construction run. Scores are normalized so that the starting solution is 0 and the target score is 1 (Eq. (1)). tackle increasingly difficult problems with less human intervention. Developing this capability requires environments that provide challenging problems and reliable feedback on attempted improvements. These environments supply experience for training and a basis for evaluation, making their availability and diversity central to the development of RSI at scale. Constructing suitable RSI environments is particularly difficult for research tasks. They must pose meaningful problems, allow continued improvement, and evaluate solutions automatically while distin- guishing genuine gains from measurement noise and scoring shortcuts. Formulating such tasks requires domain knowledge, and implementing reliable evaluators adds a separate engineering challenge. Ex- isting benchmarks draw on repositories and competitions with established tests or graders [3,12], or rely on tasks and evaluation rubrics written by experts [21,24]. These approaches provide valuable environments, but their scale and domain coverage remain tied to available evaluation resources and expert effort. This raises a central question: how can we automatically construct RSI environments at scale? We introduce RSI-Forge, a multi-agent pipeline that constructs executable RSI environments from published papers (Fig. 1). Papers provide concrete research questions, methods, and measured results that can guide environment design. An author agent develops the task, starting solution, and evaluator. A reproducer agent independently implements the paper’s method using only the paper and solver- facing materials, establishing the reproduced baseline. A reviewer agent checks that the task can be understood from its instructions and that the evaluator measures the stated objective, requesting repairs when needed. The agents have distinct responsibilities and access to information, allowing reproduction and review to assess the environment separately from the author agent’s construction. Using this pipeline, we construct 210 environments across 18 fields. To assess their quality, three agent judges evaluate the environments using a common rubric, and independent human domain experts review 90 environments within their fields of expertise. The rubric assesses whether tasks pose meaningful research problems, offer opportunities for improvement, and provide reliable evaluation. Experts assign their highest ratings to the potential for improving the starting solutions and the evaluators’ ability to distinguish solution quality. They are more critical than agent judges of shortcut resistance, fidelity to the source paper, and whether a single idea can exhaust a task (Fig. 2). 2 Scale AI Research To assess whether these environments support repeated improvement, we evaluate four solver models over 3 successive attempts on 120 of the environments. Each attempt inherits prior code and notes while model weights remain fixed. At least one model improves after the first attempt in 84% of environments, and the reproduced baseline is surpassed in 68 of the 120 environments. Transcript analysis identifies work beyond parameter tuning in 95% of these successful attempts. Analysis of the resulting trajectories further distinguishes how models explore alternatives, assess uncertain gains, and select solutions. Together, these results show that RSI-Forge can automatically construct research environments that support measurable improvement over successive attempts. Our contributions are: 1.An automated RSI environment construction pipeline. We introduce RSI-Forge, which turns pub- lished papers into executable environments. 2. A corpus spanning diverse research fields. We construct 210 environments across 18 fields, including 90 independently reviewed by human domain experts. 3.Validation through repeated improvement. We empirically demonstrate gains over successive attempts and beyond reproduced baselines, with trajectory analysis revealing differences in how models explore alternatives and assess changes. 2 Related Work Self-improving agents. Prior work studies improvement at several levels. Self-Refine iteratively revises model outputs using feedback, while Reflexion retains reflections from earlier attempts to inform subsequent decisions [15,18]. Other approaches optimize prompts, workflows, agent designs, harnesses, or source code [7,8,13,27–30]. These approaches investigate how agents improve their solutions and procedures. Our work addresses the construction of environments that supply challenging problems and measurable feedback for such improvement. Automated task and environment generation. Research on learning tasks spans curriculum learning, adaptive environment design, and autonomous task selection [2,4,11,23]. Absolute Zero and R-Zero generate tasks as part of a learning process that couples task proposal with solving [9,31]. AgentSynth synthesizes tasks and trajectories for computer-use agents [25], while EnvScaler and EnvFactory construct executable environments for tool interaction [20,26]. SPADE further couples environment generation with learning: a model designs executable environments and learns by acting within them [14]. For terminal agents, TermiGen synthesizes verifiable environments through iterative multi-agent refinement [33], while Fan et al.[6]progressively increase environment difficulty to sustain learning signals. RSI- Forge shares the goal of expanding environment supply through automation, focusing on research problems drawn from published papers across diverse fields. Research and engineering benchmarks. Existing benchmarks obtain tasks and evaluation procedures from several sources. SWE-bench uses repository issues and tests, while MLE-bench adapts machine- learning competitions [3,12]. Related benchmarks assess machine-learning experimentation and compu- tational reproducibility [10,19]. PaperBench evaluates paper replication using rubrics developed with the source papers’ authors [21]; RE-Bench provides expert-designed research engineering environments [24]; and Terminal-Bench provides curated terminal tasks with human-written solutions and verification tests [16]. RSI-Exam focuses specifically on repeated improvement in executable research tasks [1]. Closer to automated construction from papers, Theiler et al.[22]translate published methods into a shared benchmarking framework for machine health intelligence. RSI-Forge constructs research tasks and evaluators across fields, with independently reproduced baselines for measuring further improvements. 3 Scale AI Research 3 RSI-Forge RSI-Forge draws research problems and reference methods from published papers. It converts each selected problem into an executable environment and independently reimplements the paper ’s method there to establish a measured baseline. A reviewer agent checks that the instructions clearly describe the task and the evaluator measures the intended objective, requesting repairs when needed. 3.1 Environment Format We start by defining an RSI environment as an executable research task with the resources needed to develop and evaluate solutions. Each environment specifies a research objective, the required solution interface, and a compute budget. It provides a runnable starting solution, code for generating task data, and an automated evaluator. We refer to the agent attempting the task as the solver. The solver can modify or replace the starting solution and test its changes on a development set. A separate held-out test set is reserved for grading submitted solutions. Each environment includes a reproduced baseline: the source paper’s method independently reimple- mented and evaluated under the environment’s task definition and resource limits. Its measured score provides a reference for assessing improvements beyond that method. The source paper, reproduced implementation, and test data are not supplied as solver inputs. Each environment also defines a target score based on a task-specific performance bound or an idealized solution, such as one with access to information unavailable to the solver. The target serves as a normalization reference and need not be attainable with the information and resources available to the solver. To compare performance across environments, we normalize scores so that the starting solution corresponds to 0 and the target to 1. 3.2 Multi-Agent Construction Construction begins with papers whose research problems can be evaluated automatically and whose methods fit the available compute budget. For each paper, an author agent adapts the research problem into an executable task, specifying the solver’s objective, permitted changes, and resource limits. It builds the starting solution, code for generating task data, and evaluator, and prepares the development and held-out test sets. The author agent also implements the paper ’s method for preliminary checks and measures the starting-solution and target scores. The proposed environment then undergoes automated checks and review. A reviewer agent, with a context separate from the author agent’s, examines the instructions, implementation, and check results to assess whether the task can be understood from the supplied materials and whether the evaluator measures the intended research objective. Initial checks verify that the components run and that the evaluator executes the code submitted through the required interface. Scripted tests also check whether performance differences exceed measured variation across repeated evaluations. The author agent specifies how the test cases differ from the development cases—for example, by reserving different simulation settings for testing—and the reviewer agent checks this distinction. Once these checks pass, a reproducer agent receives the paper together with the task instructions, development data, and evaluator available to a solver. The author agent’s implementation of the method, construction history, the test data, and the target score are withheld. The reproducer agent independently implements the paper’s method using the same submission interface and compute budget as solver solutions. Its implementation is evaluated on the held-out test set to establish the reproduced baseline 4 Scale AI Research score. The reproduction must outperform the starting solution for the environment to pass this stage, providing direct evidence that improvement over the starting solution is possible. The reviewer agent then examines the reproduction report, and a final end-to-end test checks that the environment can start, accept a solver’s submission, and grade it. Acceptance requires successful automated checks and approval from the reviewer agent at every stage. A failed check or unresolved review finding returns the environment to the author agent for repair, and environments that still fail after the permitted repair attempts are rejected. 3.3 Scoring and Verification Scoring solver submissions. Each environment measures its research objective using a scoring program that is deterministic for fixed evaluation inputs. To enable performance comparison across environments, we normalize a raw test scorevusing the starting-solution scorewand target scoreu, both established on the same test set: b(v) = v− w u− w .(1) This maps the starting solution to 0 and the target to 1, with the reproduced baseline providing a separate comparison point on this scale. Submission validity. Alongside performance scoring, automated checks assess whether submitted solutions satisfy the required file structure, interfaces, correctness conditions, and resource limits. A model judge agent, distinct from the reviewer agent used during environment construction, also examines the submitted code for violations of task-specific rules. It assigns zero reward to a violating submission regardless of its measured performance. Preventing reward hacking. Agents may exploit access to evaluation data or tests to increase their scores without improving their solutions [17,32]. To mitigate such risk, RSI-Forge restricts access to held-out grading materials. Before submitted code runs, the evaluator loads the test data into memory and removes their on-disk copy. The solver can use development evaluations while working, but held-out scores are not returned during solving. These restrictions do not eliminate all opportunities for exploitation: some environments left other grading materials accessible. We report the observed access failures and the sensitivity of solver results to those failures in Section 5 and Appendix C.3.2. 4 Building the Corpus 4.1 Sourcing We collect recent preprints separately by domain, then deduplicate the resulting list to broaden disci- plinary coverage. Each paper is read in full against six eligibility criteria: a quantitative objective with a stated direction of improvement, a measured baseline, a metric that can be computed by a deterministic script, a method that fits the available compute budget, measurable room for improvement, and a licence permitting redistribution. All source papers are from 2026. These criteria determine which papers enter construction; they do not guarantee that an environment can be built. Appendix A.1 gives additional selection and acceptance details, and Section 4.2 reports the counts. 5 Scale AI Research Table 1. Main-run construction funnel. Counts include environments that passed after repairs. The additional 30 environments built with GPT-5.6-sol are outside this funnel. StageCount Source papers screened10,800 Papers eligible for construction2,117 Construction attempts begun213 Passed Design199 Passed Build198 Passed Independent Reproduction189 Passed End-to-End Testing180 Included in the main corpus180 Table 2. Examples of environments from the main construction run. FieldTaskEvaluation objective RoboticsChoose footholds on a terrain mapIncrease distance in six steps Life SciencesPropose molecules for a binding pocketImprove docking scores Earth, Climate & SpaceEstimate weather from partial observationsReduce state-estimation error Statistics & InferenceEstimate effects without a control seriesReduce treatment-effect error Software Engineering & Formal Methods Generate tests within a fixed budgetTrigger more distinct failures Computational PhysicsInfer a plasma boundary from magnetic dataReduce boundary-estimation error 4.2 Construction Funnel The main construction run used Claude Opus 5 for the author agent, reproducer agent, and reviewer agent, with separate sessions and role-specific inputs. Table 1 traces the source papers through eligibility and construction. We stopped after 180 accepted environments; only 213 of the 2,117 eligible papers entered this run, so the counts do not estimate the yield from all eligible papers. Among accepted environments, 79% required at least one revision after failing a construction check. The funnel therefore reports passage after any permitted repairs. Analysis of repair records identified recurring defects in task instructions and evaluation, summarized in Table 3. Construction used a median of 170 million model tokens per accepted environment. Across construction, 99.6% of input tokens were cache reads. Appendix A.2 gives the repair breakdown and construction costs. We then replaced the construction model with GPT-5.6-sol and built 30 additional environments using the same queue and checks. This yields 210 environments overall. The two construction sets are unpaired by source paper and field; the second run tests whether the procedure works with another backend, rather than estimating a paired model effect (Appendix C.4). 4.3 Corpus Composition and Contents The full corpus contains 210 environments across 18 research fields. The 180 environments from the main construction run include 6–14 environments per field. Coverage extends beyond machine learning to areas including robotics, computational physics, statistics, and software engineering. Table 2 showcases example tasks and evaluation objectives from six research fields. Table 5 reports the full field distribution. 6 Scale AI Research Table 3. Most common construction defects identified by model-based coding of repair passages. Percentages are shares of repair passages; categories overlap. Issue identifiedShare of repair passages Checks unable to detect the intended failure66% Incorrect or incomplete instructions63% Checks inspecting a different copy of the code54% Score gains without performing the intended work37% For each environment, we retain the task instructions, runnable code, code for generating task data, starting solution, reproduced baseline implementation, and evaluation configuration. Environments provide a development set for experimentation and a held-out test set for scoring. Construction records document feedback from the reviewer agent and repairs. The corpus is accompanied by quality assessments and records of agent experiments. These include human and automated ratings with written justifications, agent transcripts, saved programs, experiment logs, and held-out scores. Together, they support inspection of both the constructed environments and the sequence of changes agents make while solving them. Example environment. In one weather-estimation environment, the solver estimates a state from a small ensemble and noisy observations of half its components. The starting solution uses an ensem- ble Kalman method using raw sample covariance. The solver can test changes on 160 development instances; grading uses 324 held-out instances from different simulation conditions. The objective is to minimize mean estimation error. Held-out test errors are 1.786 for the starting solution and 1.390 for the independently reproduced paper method. With a target score of 1.156, normalization maps the starting solution to 0 and the reproduced baseline to approximately 0.629 (Eq. (1)). 4.4 Quality Three agent judges rated all 180 main-run environments, and human domain experts rated 90. Both groups used the same nine-dimension rubric, with each dimension rated from 1 to 5. On the expert- reviewed subset, the means across environments and rubric dimensions were 4.24 and 4.54 for human experts and the three-judge panel, respectively. Experts rated room above the starting solution (4.81) and discrimination between attempts (4.74) highest. These ratings support the environments’ core measurement properties (Fig. 2 and Table 6). Experts were more critical of saturation, shortcut resistance, and fidelity to the paper. The widest difference concerned whether one idea could exhaust a task: experts assigned 3.09, compared with 3.98 from the agent panel. This comparison also reflects unequal information: some experts cited the solver- result summaries withheld from agent judges. Fidelity disagreements often concerned simplifications made to fit the compute budget. Expert-review concerns are summarized in Appendix B.2.1. Removing the most generous judge lowers the panel mean to 4.39 without eliminating the largest disagreements. For the additional 30 environments built with GPT-5.6-sol, all three judges give mean ratings within a fifth of a point of their main-run ratings (Appendix C.4). Under the same rubric, agent judges rated our environments higher than manually constructed RSI-Exam tasks on distinguishing solution quality and resisting shortcuts, whereas RSI-Exam received higher ratings on resistance to saturation (Table 7). Review ratings are not used to select or weight solver outcomes. 7 Scale AI Research 12345 Mean rating (higher is better) Room above starting solution 4.814.89 Score discrimination 4.744.65 Single objective 4.614.66 Metric validity 4.404.49 Research problem 4.394.59 Development-test distinction 4.334.67 Shortcut resistance 4.094.67 Faithfulness to source paper 3.724.22 Resistance to saturation 3.093.98 Quality ratings on the same 90 environments Quality criterionRating scale: 1–5ExpertsAgent judges Human experts (one per environment)Agent judges (mean of three) Figure 2. Quality ratings from human experts and agent judges on the same 90 environments, with one expert and three judges per environment. Points show means on a 1–5 scale (higher is better). Agent ratings are averaged within environments; rows are ordered by expert mean. Only experts received available solver results. Appendix B.2.2 and Table 6 give the rubric and per-dimension standard deviations. 5 Experiments We use the environments to measure improvement over successive attempts and relative to the repro- duced baselines, and analyze how agents explore alternatives, assess changes, and select solutions. 5.1 Experimental Setup Before inspecting scores, we randomly sampled 120 environments from the main construction run, covering all 18 fields. Four solver agents use the same Claude Code CLI harness, with only the model endpoint changed: Claude Opus 5, GPT-5.6-sol, DeepSeek-V4-Flash, and Claude Haiku 4.5. Each model runs 3 successive sessions per environment, yielding 480 model–environment chains and 360 sessions per model. Sessions begin with fresh contexts. The first receives the starting solution; subsequent sessions inherit the previous session’s submitted solution, saved versions, experiment log, and notes. Sessions end when the agent stops or at 60 minutes. Almost all ended when the agent stopped. This budget is separate from each environment’s budget for running a candidate method. Models exchange no records in the main experiment, and their weights remain fixed. We report held-out scores of submitted solutions after each session, using the final submission after session 3 as the primary outcome. We also replay every saved intermediate version on the held- out test set to measure progress within sessions and assess whether agents submit their best saved solutions. Comparisons use normalized scores and within-environment ranks; bootstrap intervals resample environments. Appendix C.1 gives further protocol details, and Table 9 reports saved versions and output tokens. 8 Scale AI Research Table 4. Submitted-solution scores on 120 environments, normalized to the starting solution (0) and target (1). Session 3 confidence intervals use bootstrap resampling of environments. Mean rank averages within-environment ranks at session 3 (1 is best). Median normalized score↑Session 3 ModelSession 1 Session 2 Session 395% CIMean rank↓ Claude Opus 50.7540.7910.800[0.753, 0.867]1.67 GPT-5.6-sol0.6730.7470.761[0.723, 0.824]1.73 DeepSeek-V4-Flash0.5500.5940.647[0.586, 0.711]2.97 Claude Haiku 4.50.1290.2770.332[0.203, 0.451]3.63 Improvement Exploration Shortcuts 87868277 98817749 46728182 0 50 100 Opus 5 GPT-5.6-sol DeepSeek V4-Flash Haiku 4.5 Figure 3. Transcript-coded behavior across 360 sessions per model (120 environments, three sessions each). Cell values and color intensity show percentages of sessions assigned to each group by model-based open coding. Groups overlap; “improvement” covers developing and checking solutions, without requiring score gains. 5.2 Performance and Continued Improvement Table 4 reports submitted scores after each session. The two strongest models have overlapping median- score intervals; both outperform DeepSeek-V4-Flash, which outperforms Claude Haiku 4.5. Within- environment ranks give the same ordering, and paired bootstrap comparisons separate five of the six model pairs. These results describe performance on this corpus under the common harness. The same solver ordering holds on the 30 environments constructed with GPT-5.6-sol, although the two construction sets are unpaired by source paper and field (Table 12). Most score improvement occurs in the first session for the three stronger models, with smaller gains in later sessions. With a 0.01 threshold on gains in normalized score, at least one model improves after the first session in 84% of environments. In 68 of the 120 environments, at least one model surpasses the reproduced baseline in one of its 3 sessions. These comparisons establish progress within the constructed tasks and against their reproduced baselines (Appendix C.2). Final solution selection. Some final submissions score below an earlier saved version from the same agent, even when the better version passes the automated submission checks. Replaying intermediate versions exposes this difference between discovering a solution and selecting it. Access to held-out material.The transcript coding records access to held-out material or another run’s graded record in 22 of 480 chains. We retain these chains in the main results. Excluding them changes each model’s median normalized score by at most 0.006 and preserves the ordering (Appendix C.3.2). 9 Scale AI Research 5.3 A Three-Session Example The aggregate scores in Table 4 summarize performance across 120 environments. Figure 4 shows one Claude Opus 5 chain in detail: every saved solution graded over three successive sessions and the changes described in the agent’s records. In this environment, a humanoid crossing stepping stones, pillars, stairs, and gaps must choose its next foothold. A submitted solution receives a local elevation map, a six-component state, and a list of admissible locations, and returns the index of one location. The evaluator measures the distance reachable from that foothold within six steps, using held-out terrain generators. The submitted normalized score rises from 0.15 in session 1 to 0.33 in session 2 and 0.58 in session 3. The second session evaluates each foothold by several of its best continuations, replacing the first session’s single-best-continuation rule. In the third session, the agent adds a check that a foothold has sufficient surrounding ground support, removing invalid continuations. With these removed, it returns to the single-best-continuation rule. The trajectory illustrates how an agent can improve by revisiting an inherited design choice. 123456789101112131415 saved versions 0.00 0.25 0.50 0.75 1.00 normalized score Session 1 Replaces the starting rule with a six-step look-ahead: alternating feet, progress measured past the landing, and a preference for spots with several ways onward. Session 2 Judges a landing by several of its best continuations, not just the single best, and drops the hand-set constant. Session 3 Requires solid ground around a landing spot. That removes routes the earlier versions allowed, which makes session 2's averaging harmful, so it returns to the single best. 0.15 0.33 0.58 Figure 4. A three-session Claude Opus 5 chain on a foothold-selection task. Open points are saved solutions graded on the held-out test set; filled points mark final submissions, with normalized scores of 0.15, 0.33, and 0.58. The annotations summarize the changes recorded in each session. The third session adds a ground-support check and reverses the second session’s averaging rule. 5.4 Transcript Analysis We analyze the records to characterize how agents work. A model-based coding procedure groups practices into improvement (practices for developing and checking solutions), exploration (tests of whether a change is useful), and shortcuts (score-directed behavior without sufficient evidence of underlying improvement). For the analysis shown in Fig. 3, Claude Opus 5 identifies recurring practices without a predefined category list. Each positive assignment requires supporting quotations from the transcript. Humans did not label these trajectories (Appendices C.3 and C.3.2). Figure 3 summarizes the prevalence of the three behavior groups across models. All four models frequently modify their solutions, but their evidence-gathering practices differ. Exploration is more 10 Scale AI Research common for Claude Opus 5 than for Claude Haiku 4.5, while fitting to the development score follows the reverse pattern. The middle models exchange positions on individual practices. We check these patterns using a separate analysis with a fixed codebook of ten practices. Three additional model coders apply this codebook to a stratified sample of 30 sessions per solver model, obtaining agreement ofκ =0.73 with the original coder and recovering the same contrast in exploration and shortcuts between Claude Opus 5 and Claude Haiku 4.5 (Appendix C.3.2). A further check uses numeric logs from 139 sessions with recoverable scores and standard errors. Gains smaller than the reported standard error account for 73% of recorded accepted improvements for Claude Haiku 4.5, compared with 44% for Claude Opus 5 (Appendix C.3.3). Among sessions that outperform the reproduced baseline, 95% are coded as involving work beyond parameter tuning; 5% are tuning-only. This coding characterizes the work performed in these tasks without establishing novelty to the source paper’s field. Figure 5 reports mechanism frequencies, and Appendix C.2 gives examples of method changes. 6 Discussion Our results show that published papers can be a practical source of executable research environments across diverse disciplines. Most accepted environments required repairs before passing the construction checks (Section 4.2). Recurring defects included incorrect instructions and checks that could miss failures, showing that automated environment construction involves substantial verification and repair. The quality reviews clarify what this construction process achieves. Experts rated room above the starting solution and the evaluator ’s ability to distinguish attempts most highly, supporting the properties needed to measure progress. Their lower ratings for resistance to saturation and fidelity to the source paper identify a harder challenge: preserving the depth of a research problem when making it executable under a fixed budget. The disagreement with automated judges suggests that targeted expert review remains valuable for assessing these properties as construction scales. The solver experiments demonstrate both improvement over successive attempts and gains beyond reproduced baselines. Gains are nevertheless concentrated in the first session for the stronger models. Diminishing gains do not by themselves establish task saturation; they may also reflect limits in how agents search for alternatives or use inherited records. The trajectories also reveal weaknesses in how agents conduct experiments and retain their results. Lower-scoring models explored less and more often accepted gains smaller than the reported standard error. Across models, some final submissions were worse than previously saved versions, showing that finding a better solution does not ensure it is retained. These observations make experimental judgment and final selection concrete targets for future interventions. The saved programs, measurements, and handover records provide a basis for evaluating such interventions; the present associations do not establish which practices cause higher performance. Limitations. Tasks are adapted to bounded compute, so surpassing a reproduced baseline does not establish an advance on the original research problem. Our sessions measure progress on individual tasks; the study does not isolate the effect of inherited records from extra solving time. No trajectories were used for training; the study does not establish broader gains in research capability. Some environments left held-out grading materials accessible during solver runs. Expert review covers 90 of 180 main-run environments, with one expert each and no human inter-rater estimate; model-generated trajectory 11 Scale AI Research labels lack human validation. We do not ablate construction roles, leaving their individual contributions unresolved. Access restrictions limit independent replication of the full corpus (Appendix D). 7 Conclusion We introduced RSI-Forge, a multi-agent pipeline that turns published papers into executable RSI environments with automated evaluators and independently reproduced baselines. The corpus contains 210 environments across 18 fields. Human expert reviews highlight opportunities to improve starting solutions and the evaluators’ ability to distinguish solution quality, while identifying challenges in shortcut resistance and fidelity to the source paper. Solver experiments demonstrate gains over successive attempts and beyond reproduced baselines. Trajectories further reveal differences in how agents explore alternatives, assess uncertain gains, and select solutions. RSI-Forge provides a scalable approach to expanding the supply of research environments for developing and evaluating self-improving agents. References [1]Aiming Lab. RSI-Exam: Benchmarking recursive self-improvement through executable research. https://rsi-exam.ai/, 2026. [2]Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning, 2009. [3]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, et al. MLE-bench: Evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations, 2025. [4] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Advances in Neural Information Processing Systems, 2020. [5]Yi Duan, Ying Liu, Zirui Tang, Haodong Chen, Jun Zhou, Yumou Liu, Bangrui Xu, Yukai Wu, Sidi Chen, Yuhan Zhou, Haoyu Wang, Xiaoyou Yu, Shaokun Han, Xuzhou Zhu, Le Zhou, Bolin Lu, Wei Zhou, Jiachen Liu, Nuozhou Fang, Jiaxin Tian, Ruoyu Chen, Yuxuan Li, Kai Zuo, Kaiyan Zhang, Qianyu Yang, Zijie Wang, Jiantao Qiu, Conghui He, Guoliang Li, Bowen Zhou, Zhiyuan Liu, Zhoufutu Wen, Jihua Kang, Xuanhe Zhou, and Fan Wu. The last AI built by humans: Toward genuine recursive self-improvement, 2026. [6]Zhiyuan Fan, Tinghao Yu, Yuanjun Cai, Jiang Zhou, Jiangtao Guan, Jincheng Liu, Yun Yang, Dingxin Hu, Zhuo Han, Xing Wu, Feng Zhang, and Lilin Wang. Environment evolution for terminal agents, 2026. [7]Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution, 2023. [8] Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems, 2024. [9] Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-Zero: Self-evolving reasoning LLM from zero data, 2025. 12 Scale AI Research [10] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation. In International Conference on Machine Learning, 2024. [11] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, 2021. [12] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. [13]Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Matei Zaharia, and Yujin Tang. Recursive harness self-improvement, 2026. [14]Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao, Zichen Liu, Junsu Kim, Zijian Zhou, Seungone Kim, Tongzheng Ren, Mickel Liu, Hanfei Yu, Zhaorun Chen, Weiyan Shi, Paul Pu Liang, Luke Zettlemoyer, Yejin Choi, and Natasha Jaques. SPADE: Self-play in adaptive synthetic executable environments, 2026. [15] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, et al. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, 2023. [16]Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations (ICLR), 2026. URL https://openreview.net/forum?id=a7Qa4CcHak. [17]Pradyumna Shyama Prasad, Meiri Anto, Leon Eshuijs, Julian Moncarz, Kaustubh Kislay, and Juan J. Vazquez. BAITBENCH: Measuring agent reward hacking with optional shortcuts planted in ML tasks. arXiv preprint arXiv:2608.30724, 2026. [18] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflex- ion: Language agents with verbal reinforcement learning. In Advances in Neural Information Pro- cessing Systems, 2023. URLhttps://proceedings.neurips.c/paper_files/paper/2023/hash/ 1b44b878b782e6954cd888628510e90-Abstract-Conference.html. [19]Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE- Bench: Fostering the credibility of published research through a computational reproducibility agent benchmark, 2024. [20]Xiaoshuai Song, Haofei Chang, Guanting Dong, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. EnvScaler: Scaling tool-interactive environments for LLM agent via programmatic synthesis. In Findings of the Association for Computational Linguistics: ACL 2026, 2026. doi: 10.18653/v1/2026. findings-acl.407. URL https://aclanthology.org/2026.findings-acl.407/. [21]Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, et al. PaperBench: Evaluating AI’s ability to replicate AI research, 2025. [22]Raffael Theiler, Ludovico Comito, David Leko, Leandro Von Krannichfeldt, Lev Telyatnikov, and Olga Fink. From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence. arXiv preprint arXiv:2605.28371, 2026. [23] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, et al. Voyager: An open- ended embodied agent with large language models, 2023. 13 Scale AI Research [24] Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, et al. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts, 2024. [25]Jingxu Xie, Dylan Xu, Xuandong Zhao, and Dawn Song. AgentSynth: Scalable task generation for generalist computer-use agents, 2025. [26]Minrui Xu, Zilin Wang, Mengyi DENG, Zhiwei Li, Zhicheng Yang, Xiao Zhu, Yinhong Liu, Boyu Zhu, Baiyu Huang, Chao Chen, Heyuan Deng, Fei Mi, Lifeng Shang, Xingshan Zeng, and Zhijiang Guo. EnvFactory: Scaling tool-use agents via executable environments synthesis and robust RL, 2026. [27]Xunjian Yin, Xinyi Wang, Liangming Pan, Li Lin, Xiaojun Wan, and William Yang Wang. Gödel agent: A self-referential agent framework for recursive self-improvement, 2024. [28]Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (STOP): Recursively self-improving code generation. In Conference on Language Modeling (COLM), 2024. [29]Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Gödel machine: Open-ended evolution of self-improving agents, 2025. [30]Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation, 2024. [31]Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data, 2025. [32]Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. ImpossibleBench: Measuring LLMs’ propensity of exploiting test cases. arXiv preprint arXiv:2510.20270, 2025. [33]Kaijie Zhu, Yuzhou Nie, Yijiang Li, Yiming Huang, Jialian Wu, Jiang Liu, Ximeng Sun, Zhenfei Yin, Lun Wang, Zicheng Liu, Emad Barsoum, William Yang Wang, and Wenbo Guo. TermiGen: High-fidelity environment and robust trajectory synthesis for terminal agents, 2026. 14 Scale AI Research A Construction Details A.1 Selection and Acceptance The six sourcing criteria are given in Section 4.1. A task must provide a single ranking of solutions; additional quantities may impose fixed constraints, but unconstrained trade-offs between objectives do not meet this requirement. Screening for room to improve does not establish that the eventual target score is attainable. The source paper’s reported performance guides construction but does not replace measurement of the reproduced method within the adapted environment. The roles and information restrictions are described in Section 3. All three construction roles use Claude Opus 5 in separate contexts in the main run; this separation does not establish that their errors are independent. Acceptance requires automated checks and reviewer-agent approval at every stage. Development and test sets must differ along a documented task condition, rather than only by a random partition. Independent reproduction uses the same execution path and container as solver solutions and must outperform the starting solution. The author agent has at most 3 repair attempts per stage before rejection. End-to-end checks cover startup, submission, and grading; they do not establish that all grading materials are inaccessible (Appendix C.3.2). A.2 Repairs and Cost Among the 180 accepted main-run environments, 79% required repair: 53% during design, 45% during build, 21% during independent reproduction, and 20% during end-to-end testing. These stage shares overlap. Main Table 3 summarizes model-coded defects in author-agent repair passages; its percentages describe passages, not environments, and categories overlap. Median construction cost is 170 million model tokens per accepted environment. Median per-environment shares are 82%, 14%, and 3% for author, reviewer, and reproducer, respectively. These separately summarized medians need not sum to 100%. B Corpus Composition and Quality B.1 Field Coverage Table 5 reports field counts for the 180 main-run environments; Appendix C.4 describes the additional 30 environments constructed with a second backend. Table 5. Field composition of the 180 main-run environments. FieldCountFieldCount Robotics14Security & Cryptography10 Life Sciences13Neuroscience9 Earth, Climate & Space13Optimisation & Discrete Algorithms9 Statistics & Inference13Chemistry & Materials9 Networks & Information Th.12Graphics & Rendering8 Software Eng. & Formal Methods12Quantitative Finance7 Signal, Imaging & Control11Computational Physics7 Computer Systems10Mechanical & Thermal7 Numerical & Sci. Computing10Machine Learning6 15 Scale AI Research B.2 Review Protocol and Results Three agent judges rated all 180 main-run environments: Claude Opus 5 under Claude Code, GPT-5.6-sol under Codex at high reasoning effort, and Gemini 3.7 Flash under Antigravity. Each model–harness pair is one judge. Judges read task instructions and evaluators and can run code, with network access for field knowledge; the author agent’s construction transcript, held-out test data, and solver-result summaries are withheld. Domain experts independent of environment construction, each holding a doctorate in the relevant field, reviewed 90 environments, with one expert per environment and a written justification for every rating. All reviews enter the analysis; there is no human inter-rater estimate. Both groups use the same 9-dimension, 1–5 rubric (Table 8) and have network access. Experts receive the source paper after submitting ratings for the eight environment-focused dimensions, then rate faithfulness last. Only experts receive available summaries of solver scores over successive sessions. Neither group receives construction-stage verdicts or reviewer-agent notes. Expert–agent comparisons therefore reflect differences in evidence as well as judgment. Harness sensitivity. For Gemini 3.7 Flash, 1,620 paired dimension ratings on 180 environments have a mean absolute difference of 0.13 points between Claude Code and Antigravity; 99% differ by at most one point. Mean ratings are 4.88 and 4.82, respectively. This indicates limited sensitivity for this judge under these two harnesses. Table 6 gives matched expert and agent-panel ratings with standard deviations. Removing Gemini 3.7 Flash, which gives the highest mean on every dimension, lowers the panel mean from 4.54 to 4.39, compared with 4.24 for experts. Resistance to saturation and shortcut resistance remain the two largest positive gaps (+0.69 and +0.42, respectively). Ratings are not used to select or weight solver outcomes. Table 6. Human-expert and agent-panel ratings on the same 90 environments. Scores are means on a 1–5 scale; SD denotes standard deviation. Agent ratings are averaged within environments; Gap is the panel mean minus the expert mean. DimensionExpertsSDPanelSDGap Research problem4.390.844.590.32 +0.20 Room above starting solution4.810.514.890.31 +0.08 Resistance to saturation3.090.963.980.84 +0.89 Faithfulness to source paper3.720.874.220.51 +0.50 Metric validity4.400.854.490.45 +0.09 Single objective4.610.904.660.46 +0.05 Shortcut resistance4.090.974.670.55 +0.58 Score discrimination4.740.624.650.55 −0.09 Development–test distinction4.330.934.670.49 +0.34 All nine dimensions4.240.994.540.58 +0.30 B.2.1 Expert–Agent Disagreements The largest agent-panel minus expert gaps concern resistance to saturation (+0.89), shortcut resistance (+0.58), and faithfulness (+0.50). Expert reviews illustrate these concerns. Resistance to saturation. One expert gave the lowest rating after a supplied search method covered about 69% of the declared improvement band in roughly five CPU-minutes. The expert also cited solver results reaching the target score in the first session, which agent judges did not receive. The disagreement 16 Scale AI Research concerns both available evidence and interpretation of the rubric: reaching a normalization target does not rule out further raw gains or establish that the research problem is exhausted. Shortcut resistance. One expert proposed adapting SQL answers through repeated held-out grading feedback. The intended protocol withholds that feedback and removes test files before submitted code runs (Section 3.3); whether this attack is possible depends on runtime access controls, not task wording alone. Separately, another review found supplied parameters and held-out family definitions that would let a solution branch on those definitions instead of estimating the intended quantity. We treat that exposure as an environment defect. Observed solver access failures are reported in Section 5.2 and Appendix C.3.2. Faithfulness. An expert assigned 2 against the panel’s 4.5: the task retained 50-way zero-shot clas- sification and comparison of audio and text semantics, but replaced real THINGS-EEG recordings, pretrained feature pipelines, and the trimodal BraVL VAE with synthetic low-dimensional arrays and a linear-Gaussian brain model. A bounded-compute adaptation can preserve the broad question while omitting substantive parts of the source method. Table 7. Ratings from the same three agent judges on 34 manually constructed RSI-Exam tasks and the 90 RSI-Forge environments in Table 6. Gap is RSI-Forge minus RSI-Exam; the 8 dimensions exclude faithfulness because this comparison has no common source-paper reference. The task sets are unpaired, so the differences do not isolate the effect of automated construction. DimensionRSI-ExamRSI-ForgeGap Score discrimination2.644.65 +2.01 Development–test distinction3.234.67 +1.44 Shortcut resistance3.924.67 +0.75 Single objective4.644.66 +0.03 Room above starting solution4.874.89 +0.02 Research problem4.614.59 −0.02 Metric validity4.574.49 −0.08 Resistance to saturation4.413.98 −0.43 all 84.114.58 +0.46 B.2.2 Review Rubric Human experts and agent judges use the same quality-review rubric, separate from construction accep- tance checks and submission-validity checks. Table 8 reproduces the descriptions for scores 1, 3, and 5. Terms such as “starter,” “ceiling,” and “split” are retained from the rubric wording; in the surrounding text we use starting solution, target score, and development/test sets. 17 Scale AI Research Table 8. Descriptions of scores 1, 3, and 5 in the environment-quality rubric, reproduced verbatim. The dimensions are grouped here as in the rating summaries; the human review form presents faithfulness to the source paper last. AxisA score of 1A score of 3A score of 5 Real problema toy with no research contenta real problem, narrowly posedyou would be pleased to see a student work on this Headroomthe starter is already near the ceiling a competent effort gains some- thing clearly large, and you can name where it comes from Not saturable one change takes most of the range the first change is worth it, more remains improvement requires several independent ideas Faithful to the paper unrelated to what the paper does related, but the hard part has been removed this is the paper’s problem, honestly posed Metric validitymeasures something elsea reasonable proxythis is the quantity the field would report Single directiontwo objectives silently tradedone quantity, the constraint is loose one quantity, one direction, the constraint binds Resistant to shortcuts you can name a specific trick and say why it scores you suspect one exists but cannot name it you looked and the scorer closes the routes you tried Discriminating nothing states it and the metric looks noisy stated, or inferable, but not justified stated with the room to im- prove, and the room is the larger of the two Split is a real axisa uniform random partition dressed up as an axis an axis, mild shift a real axis; doing well on it means something C Additional Experimental Details C.1 Experimental Setup The protocol in Section 5.1 uses solver containers with network access restricted to the model endpoint. The saved files described there are the only inheritance between sessions: the live workspace, conversa- tional context, and shell history are not carried over. Development-evaluator calls have no fixed count limit. The final submission is the solution left in the designated method directory when the agent stops or the 60-minute session ends; it is inherited regardless of its score. This session limit is separate from each environment’s compute budget for running a candidate method. Every saved version, including abandoned versions, and the final submission are replayed on held-out data. These scores are not returned during solving. A session without a valid measurement remains unscored rather than receiving an imputed score. Table 9 summarizes saved versions and output volume for the main sample. Starting and target scores. In Eq. (1), the starting-solution scorewmaps to 0 and the targetuto 1; better-than-target scores exceed 1. Construction records identify the data, code, or bound supporting each value and mark provisional values. Targets may use privileged information, an information- theoretic or optimization bound, exact solutions on tractable instances, or a Bayes rate. They need not be attainable, and normalization does not validate them. If the reproduced baseline is too close to the target, construction checks require revision of the target or task conditions. 18 Scale AI Research Table 9. Activity per session on the 120 main evaluation environments. Versions are saved runnable solutions that were graded; output tokens are in thousands. Means include one standard deviation. Token volume is not equivalent to monetary or computational cost across providers. Versions gradedOutput tokens (thousands) ModelMedianMeanMedianMean Claude Opus 54.03.6± 1.669.372.2± 22.8 GPT-5.6-sol2.02.7± 1.722.122.6± 8.3 DeepSeek-V4-Flash2.02.1± 1.154.553.8± 16.3 Claude Haiku 4.52.02.7± 1.930.229.8± 8.8 C.2 Improvements over Reproduced Baselines At least one session surpasses the reproduced baseline in 68 of 120 environments. The per-model counts are 65 for GPT-5.6-sol, 63 for Claude Opus 5, 54 for DeepSeek-V4-Flash, and 24 for Claude Haiku 4.5. These counts overlap, and an environment can contribute several successful sessions. The median normalized baseline score is 0.76 where no model surpasses it and 0.46 where at least one does. The separate mechanism codebook in Fig. 5 applies only to these successful sessions. Across them, 95% receive at least one of the 9 labels beyond parameter tuning, 5% receive only the tuning label, and 0% receive neither. These labels include repairing a non-running solution and evaluating alternatives, so they do not all imply a new method. Tuning can coexist with other work: it appears in 34% of successful Claude Opus 5 sessions and 77% of successful Claude Haiku 4.5 sessions. The coding does not establish which activity caused a gain or whether the result advances the source paper’s field. Actual method changes include retaining alternating least squares while replacing a non-differentiable tolerance with a smooth surrogate, and integrating over parameters that the reproduced baseline estimated at a single point. C.3 Behavior Coding and Analysis Three coding procedures serve different purposes: open coding supplies the behavior groups in Fig. 3; a fixed practice codebook checks those patterns; and the separate mechanism codebook characterizes successful sessions (Appendix C.2). All labels are model-generated, without human validation. C.3.1 Coding Procedures Open coding. Claude Opus 5 summarizes each session’s changes, evidence, score-directed work, unexpected events, and stopping decision, then groups recurring practices in bounded batches without a predefined category list. Document identifiers are retained through merges, and positive assignments require matching transcript quotations. This yields 62 families mapped to improvement (developing and checking solutions, without requiring score gains), exploration (testing whether a change is useful), and shortcuts (score-directed behavior with insufficient evidence of underlying improvement), plus one ungrouped family for submitting the inherited solution unchanged. A session receives a group label if it has at least one family in that group; labels overlap and need not sum to 100%. Fixed practices. Claude Opus 5 proposes categories from 24 documents selected with a fixed seed and no supplied category list. The ten-category codebook is then frozen and applied to all available 19 Scale AI Research 020406080 share of each model's successful sessions (%) select using evidence beyond the development set derive the parametrization from the mechanism estimate remaining room for improvement compute parameters for individual instances use an input the baseline discarded complete work left by the previous session repair a solution that would not run solve a subproblem exactly delete a stage of the method tune parameters of the baseline approach work beyond parameter tuning parameter tuning Work in sessions surpassing the reproduced baseline Claude Opus 5GPT-5.6-solDeepSeek-V4-FlashClaude Haiku 4.5 Figure 5. Transcript-coded mechanisms as percentages of each model’s sessions that surpass the reproduced baseline. The codebook was derived from a sample and fixed before counting. Sessions may receive several labels; nine categories cover work beyond parameter tuning. The labels describe work within the constructed task and do not establish scientific novelty. session records (Table 10). A session can receive multiple labels or none; assignments without a matching quotation are discarded. Quotations make decisions traceable without validating the interpretation or the narrated action. Within-model assignment counts explain little variation in output-token counts, but this does not rule out narration-style differences; additional coders and numeric logs provide complementary checks below. C.3.2 Behavior Patterns and Sensitivity Checks Shortcut labels and access failures. Common shortcut families include retaining a version because its development score rises and fitting constants to the development set. These can occur within the permitted workflow; the label concerns limited evidence of generalization. Other families cover reading answer keys, grading code, another run’s records, or reconstructing tests from descriptions. These are broader than the access-failure criterion: a family may include labels supplied with development data, whereas an access flag requires a record naming held-out material or another run’s graded record. The coding cannot exclude unrecorded access. This narrower criterion flags 22 of 480 model–environment chains. They remain in the main results. Excluding flagged chains gives median final normalized scores of 0.797, 0.761, 0.641, and 0.332 for Claude Opus 5, GPT-5.6-sol, DeepSeek-V4-Flash, and Claude Haiku 4.5, respectively: ordering is unchanged, and no median moves by more than 0.006. Filtering is by chain; a flag for one model does not exclude the other models on that environment. In 4 of the 68 environments where a model surpasses the baseline, a successful session carries an access flag. This overlap does not attribute the gain to access. Behavior-group percentages use the full sample; only final-score sensitivity uses the exclusion. 20 Scale AI Research Improvement 21 families · 2,649 assignments Claude Opus 5GPT-5.6-solDeepSeek-V4-FlashClaude Haiku 4.5 use error analysis to guide changes 28223328 review the previous session's record 19361918 select configurations by constraint margins 21141513 measure structural properties of task data 1912178 verify claims against supporting files 1761012 inspect environment code to understand the task 421811 reuse information from earlier records 92733 inspect program behavior before editing 66159 test the final submission 91645 test inherited claims 21142 measure a baseline directly 31073 derive quantities instead of fitting 74102 guide changes with mathematical derivations 7<1112 retain and re-evaluate saved solutions 37<19 detect a contaminated workspace 5<177 adapt parameters to individual instances 1366 document results for the next session 8401 diagnose problems through direct inspection <1363 prepare a runnable submission 4431 select candidates by output properties 3342 reformulate the solution mathematically 2451 Shortcuts 14 families · 2,356 assignments Claude Opus 5GPT-5.6-solDeepSeek-V4-FlashClaude Haiku 4.5 keep changes when development scores improve 11445571 fit constants to the development set 17445156 act on unverified claims 1153432 design the method around evaluator logic 116137 read packaged task files 12432 read the answer key 6666 hard-code values from development examples 2586 accept gains within measured variability <12812 reconstruct a proxy for the test set 1252<1 submit despite acknowledged overfitting 2<1106 read another run's workspace <11021 bound performance using hidden information 5251 infer hidden properties from task structure 2<184 assume test conditions without measurement 1155 Exploration 27 families · 3,223 assignments Claude Opus 5GPT-5.6-solDeepSeek-V4-FlashClaude Haiku 4.5 compare gains with measured variability 4624115 evaluate on data excluded from fitting 413165 confirm gains with another evaluation 332543 abandon approaches after negative results 3131511 verify checks can detect failures 46623 estimate potential gains before implementing 33510<1 select using a validation subset 19233<1 select using reasoning beyond scores 151278 reject gains that do not reflect improvement 196105 verify results with an independent method 23582 bound potential gains before searching 124131 investigate implausible measurements 123107 test the evaluation procedure 2234<1 run additional generalization tests 111221 override the development-score ranking 6785 perturb one factor at a time 10462 screen on fewer cases, confirm on more 31136 diagnose from parameter-sweep patterns 1222<1 select components using intermediate metrics 6272 compare gains across individual cases 5711 check measured cost against the budget 2452 identify limits of development evaluation 634<1 retain multiple alternatives 4352 guide changes with diagnostic measurements 3244 isolate contributions with ablations 9<130 revise conclusions after contrary evidence 5<142 trace anomalies to the evaluation setup 4061 62 families. Each cell shows the percentage of a model's 360 sessions assigned to that family. Shading is scaled separately within each group. Figure 6. Open-coded families grouped as improvement (21), exploration (27), and shortcuts (14). Cells show percentages of each model’s 360 sessions assigned a family. Labels overlap. Shading is scaled within each group; printed percentages support comparisons across groups. The ungrouped family for submitting the inherited solution unchanged is omitted. 21 Scale AI Research Table 10. The ten fixed practices, distinct from the open-coded families in Fig. 6 and the successful-session mechanisms in Fig. 5. Labels and operational criteria are paraphrased from the coding scheme; timing requirements and exclusions are retained. GroupPracticeRequired evidence ImprovementCheck inherited results Before any edit, rerun the inherited solution to reproduce its recorded score or independently derive its central claim. Use the task specification Justify the submitted change from a task-document fact or metric prop- erty; a parameter-sweep result alone is insufficient. Improve robustnessAddress failures such as degenerate inputs, timing limits, or fallback behavior without targeting the metric. ExplorationBound potential improvement Before searching, compute an upper or lower performance bound and decompose the remaining gap to allocate the budget. Test beyond development data Evaluate a change beyond the supplied development instances using a stress test or test of invariance. Compare with error bars Use an explicit error bar when accepting or rejecting a change; a before- and-after comparison alone is insufficient. Test controls or independent checks Check that a measurement detects the claimed effect using a deliberately weakened variant, an incorrect-generator control, or an independent computation. Reject unreliable gains Discount a gain because its evidence is circular, depends on a parameter fitted to development data, or changes sign across evaluations. ShortcutSelect by development score alone Run successive sweeps of individual parameters and treat any favorable development-score change as an improvement. NeitherEnvironment failure An environment failure unrelated to the solution method consumes the session. Additional model coders.Claude Sonnet 5, Gemini 3.7 Flash, and GPT-5.6-sol apply the fixed practice codebook to a stratified sample of 30 sessions per solver model. Agreement with the original coder is κ =0.73 over the 9 categories used by either coder. On these records, the original coder labels exploration in 97% of Claude Opus 5 sessions and 30% of Claude Haiku 4.5 sessions; additional coders give 93% and 13%. Shortcut rates reverse the contrast: 3% versus 80% originally, and 0% versus 63% with additional coders. This corroborates the contrast without establishing individual-label correctness. These rates use the fixed codebook on the sampled subset, whereas Fig. 3 uses open-coded families on all 360 sessions per model. C.3.3 Gains Smaller than the Reported Standard Error For 139 sessions, evaluator logs provide both scores and standard errors. We classify an accepted gain as smaller than the reported error when the selected version improves on the best previous version by less than the standard error attached to the selected score. This describes selection; it is not a significance test for the difference between correlated scores. Accepted improvements in these recoverable sessions form the denominator, not all sessions or saved versions. The rates are 73% for Claude Haiku 4.5 and 44% for Claude Opus 5. This agrees with their fixed-codebook ordering, while measuring a different behavior: the codebook concerns selection from development-score changes without sufficient supporting evidence. 22 Scale AI Research C.4 Environments by Backend Model Replacing Claude Opus 5 with GPT-5.6-sol in the same sourcing queue, checks, and repair procedure produces 30 additional environments, for 210 overall. The comparison uses accepted environments only and does not measure construction yield. Sets are unpaired by source paper and field, so differences can reflect task composition as well as backend choice. All three judges’ mean ratings differ by less than 0.2 points (Table 11). Solvers retain the same ordering (Table 12); this extends the observation to a second construction set without isolating a causal backend effect. Table 11. Agent-judge ratings by construction backend. Cells average over rubric dimensions, then environments; ncounts rated environments. For this comparison, Gemini 3.7 Flash uses Claude Code; Appendix B.2 compares its ratings under Claude Code and Antigravity. Claude Opus 5GPT-5.6-sol Agent judgeMeannMeann Claude Opus 5 (Claude Code)4.371804.2530 GPT-5.6-sol (Codex)4.401804.2730 Gemini 3.7 Flash (Claude Code)4.881804.7430 Table 12. Median final submitted normalized scores after three sessions by construction backend. The main- backend column repeats session 3 of Table 4;ncounts chains with all three sessions scored. Construction sets are unpaired. Claude Opus 5GPT-5.6-sol Solver modelMediannMedian n Claude Opus 50.8001200.78430 GPT-5.6-sol0.7611200.73630 DeepSeek-V4-Flash0.6471200.56430 Claude Haiku 4.50.3321200.36330 D Release and Access The corpus is available on request. Access is gated because the sealed test split and grading key make grading possible and also make scores obtainable without solving the task. We do not release the sourcing procedure, per-paper verdicts, or rejection reasons, to avoid attaching our assessments of benchmark suitability to individual preprints and their authors. 23