Paper deep dive
Evaluation-driven Scaling for Scientific Discovery
Haotian Ye, Haowei Lin, Jingyi Tang, Yizhen Luo, Caiyin Yang, Chang Su, Rahul Thapa, Rui Yang, Ruihua Liu, Zeyu Li, Chong Gao, Dachao Ding, Guangrong He, Miaolei Zhang, Lina Sun, Wenyang Wang, Yuchen Zhong, Zhuohao Shen, Di He, Jianzhu Ma, Stefano Ermon, Tongyang Li, Xiaowen Chu, James Zou, Yuzhi Xu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/26/2026, 11:13:16 PM
Summary
The paper introduces SIMPLETES (Simple Test-time Evaluation-driven Scaling), a framework designed to scale scientific discovery by optimizing the evaluation-driven discovery loop. Unlike traditional test-time scaling that focuses on generation-side compute (like reasoning tokens), SIMPLETES scales along three dimensions: global width (parallel exploration), refinement depth (feedback-driven refinement), and local sample size (selection). The framework was tested across 21 scientific problems in six domains, including Quantum Circuit Compilation, GPU Kernel Optimization, and Mathematics. Results show that SIMPLETES, using open-source models, consistently achieves state-of-the-art (SOTA) solutions, outperforming frontier models and specialized optimization pipelines. Additionally, the authors demonstrate that post-training on successful trajectory-level histories enables models to generalize discovery capabilities to unseen problems.
Entities (9)
Relation Signals (5)
SIMPLETES → addresses → Evaluation-driven Scaling
confidence 100% · We introduce Simple Test-time Evaluation-driven Scaling (SimpleTES), a general framework that strategically combines parallel exploration, feedback-driven refinement, and local selection
SIMPLETES → discoverssotain → Quantum Circuit Compilation
confidence 100% · SIMPLETES delivers strong cross-platform gains, discovering policies that surpass hand-engineered baselines
SIMPLETES → discoverssotain → GPU Kernel Optimization
confidence 100% · On GPU kernel optimization problems, SIMPLETES delivers strong results across both high-level operators and lower-level primitives.
SIMPLETES → discoverssotain → Erdős Minimum Overlap
confidence 100% · For the Erdős problem, by employing a coarse-to-fine optimization pipeline, SIMPLETES found a construction achieving 0.380856.
SIMPLETES → improves → LASSO algorithm
confidence 100% · Particularly, we sped up the widely used LASSO algorithm by over 2x
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Language models are increasingly used in scientific discovery to generate hypotheses, propose candidate solutions, implement systems, and iteratively refine them. At the core of these trial-and-error loops lies evaluation: the process of obtaining feedback on candidate solutions via verifiers, simulators, or task-specific scoring functions. While prior work has highlighted the importance of evaluation, it has not explicitly formulated the problem of how evaluation-driven discovery loops can be scaled up in a principled and effective manner to push the boundaries of scientific discovery, a problem this paper seeks to address. We introduce Simple Test-time Evaluation-driven Scaling (SimpleTES), a general framework that strategically combines parallel exploration, feedback-driven refinement, and local selection, revealing substantial gains unlocked by scaling evaluation-driven discovery loops along the right dimensions. Across 21 scientific problems spanning six domains, SimpleTES discovers state-of-the-art solutions using gpt-oss models, consistently outperforming both frontier-model baselines and sophisticated optimization pipelines. Particularly, we sped up the widely used LASSO algorithm by over 2x, designed quantum circuit routing policies that reduce gate overhead by 24.5%, and discovered new Erdos minimum overlap constructions that surpass the best-known results. Beyond novel discoveries, SimpleTES produces trajectory-level histories that naturally supervise feedback-driven learning. When post-trained on successful trajectories, models not only improve efficiency on seen problems but also generalize to unseen problems, discovering solutions that base models fail to uncover. Together, our results establish effective evaluation-driven loop scaling as a central axis for advancing LLM-driven scientific discovery, and provide a simple yet practical framework for realizing these gains.
Tags
Links
- Source: https://arxiv.org/abs/2604.19341v1
- Canonical: https://arxiv.org/abs/2604.19341v1
Trouble viewing inline? Open PDF directly →
Full Text
382,300 characters extracted from source content.
Expand or collapse full text
Evaluation-driven Scaling for Scientific Discovery Wizard Intelligence Learning Lab, Stanford University Peking University, Tsinghua University The Hong Kong University of Science and Technology (Guangzhou) Homepage Abstract Language models are increasingly used in scientific discovery to generate hypotheses, propose candi- date solutions, implement systems, and iteratively refine them. At the core of these trial-and-error loops liesevaluation: the process of obtaining feedback on candidate solutions via verifiers, simulators, or task- specific scoring functions. While prior work has highlighted the importance of evaluation, it has not explicitly formulated the problem ofhow evaluation-driven discovery loops can be scaled up in a principled and effective manner to push the boundaries of scientific discovery, a problem this paper seeks to address. We in- troduceSimple Test-timeEvaluation-drivenScaling (SIMPLETES), a general framework that strategically combines parallel exploration, feedback-driven refinement, and local selection, revealing substantial gains unlocked by scaling evaluation-driven discovery loops along the right dimensions. Across 21 scientific problems spanning six domains, SIMPLETES discoversstate-of-the-artsolutions using gpt-oss models, consistently outperforming both frontier-model baselines and sophisticated optimization pipelines. Particularly, we sped up the widely used LASSO algorithm by over2, designed quantum circuit routing policies that reduce gate overhead by 24.5%, and discovered new Erdős minimum over- lap constructions that surpass the best-known results. Beyond novel discoveries, SIMPLETES produces trajectory-level histories that naturally supervise feedback-driven learning. When post-trained on suc- cessful trajectories, models not only improve efficiency on seen problems but also generalize to unseen problems, discovering solutions that base models fail to uncover. Together, our results establish effec- tive evaluation-driven loop scaling as a central axis for advancing LLM-driven scientific discovery, and provide a simple yet practical framework for realizing these gains. Scaling evaluation-driven loop Scaling reasoning compute STUCK R= 0.9438 deep tuning in a single direction STUCK R= 0.9210 can't interpret feedback R= 0.9352 multi-res relay R= 0.9150 marginal gain R= 0.9378 pop. Adam + softmax [optimizer] R= 0.9414 spectral warm-start [initialization] R= 0.9488 FFT conv gradient [gradient] R= 0.9610 res 4K → 65K [resolution] NEW SOTA R= 0.9627 breakthrough across axes SimpleTES (Ours) reasoning-centric evaluation-centric Discovery: finding functions that maximize a self-convolution ratio Quantum Circuit Compilation superconducting_qubit_routingmin prev best human/AI60,189 SimpleTES45,441 GPU Kernel Optimization trimul_H100 (ms)min prev best human/AI 1.131 SimpleTES1.122 Algorithm Engineering lasso_path (ms)min prev best human/AI4,139 SimpleTES2,502 Mathematics Extremal Analysis erdos_min_overlapmin prev best human/AI0.380871 SimpleTES0.380856 Combinatorial Construction sum_difference_problemmax prev best human/AI1.121936 SimpleTES1.144887 Data Science scaling_law_discovery_r2max prev best human/AI0.572 SimpleTES0.674 arXiv:2604.19341v1 [cs.LG] 21 Apr 2026 Contents 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 1.1 Highlights on Discoveries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Test-Time Evaluation-driven Scaling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 2.1 Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 SIMPLETES. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 Implementation Details. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4 Learning to Scale Evaluation-driven loop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3 Scientific Discovery Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 3.1 Quantum Circuit Compilation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.1.1 Qubit Routing on Superconducting Quantum Computer. . . . . . . . . . . . . . . . 15 3.1.2 Compilation for Zoned Neutral Atom Quantum Architecture. . . . . . . . . . . . . 19 3.2 GPU Kernel Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2.1 TriMul. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.2.2 Batched Cumsum. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.2.3 Asymmetric Matmul. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3 Algorithm Engineering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3.1 Lasso Regularization Path. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3.2 AtCoder Heuristic Contests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.4 Mathematics Extremal Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.4.1 Erdős Minimum Overlap. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.4.2 Autocorrelation Inequalities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.5 Combinatorial Construction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.5.1 Sum-Difference Problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.5.2 Circle Packing in a Unit Square. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.5.3 Hadamard Maximum Determinant. . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.6 Data Science. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.6.1 Scaling Law Discovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.6.2 Single-Cell RNA-Seq Denoising. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4 Method Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46 4.1 Experiments on the Scaling Behavior of SIMPLETES. . . . . . . . . . . . . . . . . . . . . . . . 46 4.2 Experiments on Post-Training. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.3 From Golden Metrics to Surrogate: Hacking Analysis. . . . . . . . . . . . . . . . . . . . . . 50 4.4 Ablation Studies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.4.1 Ablations on Different Designs ofΦ. . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.4.2 Ablation on Reflection and Failure Patterns. . . . . . . . . . . . . . . . . . . . . . . . 54 4.4.3 Efficiency Analysis: Trajectory-level Pruning. . . . . . . . . . . . . . . . . . . . . . . 55 5 Related Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56 5.1 Existing Evaluation-Driven Discovery Methods. . . . . . . . . . . . . . . . . . . . . . . . . . 56 5.2 Self-evolving AI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 5.3 LLM for Scientific Discovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.4 Test-Time Scaling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 6 Limitation & Future Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61 7 Appendix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72 2 Evaluation-driven Scaling for Scientific Discovery 1Introduction Scientific discoveries areopen-ended: progress toward unknown high-quality solutions typically requires repeated cycles of research, proposal, experiment, and refinement [64,89]. Whether the objective is to find new mathematical constructions [9,16,46,82,92,95], engineer high-performance GPU kernels [1,12,19], optimize quantum circuits [66,70,132], or discover new biological mechanisms [10,71,77,146], cycles of trial-and-error guided by external feedback, often from physical or computational experiments, are the foundation of breakthroughs. In this cycle of scientific development, large language models (LLMs) are becoming increasingly capable, with powerful models and agentic systems being widely used for idea generation [8,75,117,129], system implementation [59,110,135,160], solution revision [79,113,153], paper writing [52,75,110,128,159], and more. Given a problem description and prior attempts, recent works [38,89,105,166] have scaled test- time compute to propose novel candidates that would take humans significant effort to conceive [123,157, 170]. These approaches iteratively generate candidate solutions and obtain feedback from an evaluator, an indispensable component of scientific discovery [75,89,105,114], forming anevaluation-driven discovery loopthat uses external feedback to guide subsequent refinement. Regardless of the diverse designs in these systems, the scaling effect of the evaluation-driven discovery loop itself remains underexplored. Existing methods either focus primarily on scaling generation-side computation, such as reasoning tokens [ 84,123,157] or agent turns [61,73,175], or aim to improve results with only limited rounds of discovery loops [72,100]. Yet this loop is precisely the mechanism through which science advances: one round of attempts produces the feedback that shapes the next. This leads to the central question of the paper:how far can scientific discovery be pushed by effectively scaling evaluation- driven discovery loops at test time? This paper introduces SIMPLETES, an algorithmic framework designed to answer the question. SIMPLETES studies evaluation-driven discovery through a compact and explicit policy space that isolates the core dimensions along which the loop can be scaled. Through theoretical modeling and empirical analysis, we identify three factors: global width for parallel exploration, refinement depth for feedback-driven re- finement, and local sample size for local selection. By combining these dimensions, SIMPLETES yields substantial gains even in settings where existing heuristic approaches fail to achieve, suggesting that for scientific discovery, scaling the evaluation-driven loop in a principled manner can be as important as scaling model capability or generation-side computation. Across 21 open-ended problems that span six domains on mathematics extremal analysis, combinatorial construction, GPU kernel optimization, quantum circuit compilation, algorithm engineering, and data science, SIMPLETES discovers novel state-of-the-art solutions using only open-source gpt-oss models. It consistently outperforms existing approaches that rely on more powerful frontier models, on ensembles of multiple frontier models, or on significantly more complex optimization pipelines, highlighting the practical importance of systematically scaling evaluation-driven discovery loops. Moreover, we conduct comprehensive ablations and scaling studies, clarifying how each component of SIMPLETES contributes to discovery performance across domains. Furthermore, SIMPLETES produces structured, trajectory-level histories that are natural supervision for feedback-driven discovery. To test whether they can teach models how evaluator feedback should shape subsequent refinement, we post-train the model on successful trajectories, assigning supervision based on the best outcome achieved within each trajectory rather than on the immediate score of each solution. Surprisingly, the resulting model achieves more efficient discovery on all training problems and transfers this capability to unseen out-of-distribution problems, finding stronger solutions than vanilla models can uncover. This suggests that evaluation-driven histories induce generalizable discovery behaviors beyond what test-time scaling alone provides. 1.1Highlights on Discoveries Quantum Circuit Compilation.Quantum circuit compilation maps logical circuits onto physical hard- ware while minimizing the execution overhead required to satisfy hardware constraints. SIMPLETES de- livers strong cross-platform gains, discovering policies that surpass hand-engineered baselines in both major hardware settings: on superconducting architectures, it outperforms the gold-standard SABRE al- gorithm [ 66] and its improved modern variant LightSABRE [176] by 21.7% and 14.9%, respectively, across Page 3 of110 Evaluation-driven Scaling for Scientific Discovery Te s t-timeEvaluation-drivenLoop Scaling Scaling Across C×퐿×K 퐶 L ... ... ... ......... global width C refinement depthL local sample size K evaluator selection K Generator G proposal x " solution y # #$% & append argmax ' ( ) ! ) Policy휋 Evaluator V Ta s k 푥 ! Tr a j e c t o r y c 푙표푝×퐿 푥 ! Scientific Tasksacross Domains score v ( y # ) Erdős Minimum Overlap cp_figure_preview Circle Packing in a Unit Square QuantumCircuit Compilation ×2 tasks GPU Kernel Optimization ×3tasks Algorithm Engineering ×3tasks Mathematics Extremal Analysis ×4tasks Combinatorial Construction ×4tasks Data Science ×5 tasks Qubit RoutingTriangle Multiplicative Update Lasso Regularization Path Single-Cell RNA-Seq Denoising Figure 1:Overview of SIMPLETES. (a)SIMPLETES scales the evaluation-driven discovery loop by allocating the evaluator-query budgetN=CLKacross three dimensions. Global widthCcontrols the number of independent trajectories, refinement depthLcontrols the depth each trajectory iteratively generates new solution candidates based on historically accumulated feedback, and local sample sizeKcontrols multiple candidates from the same proposal at each refinement step and commits only the highest-scoring candidate as the next node.(b)Task coverage: 21 scien- tific discovery tasks organized into six domains, including mathematics extremal analysis, combinatorial construction, GPU kernel optimization, quantum circuit compilation, algorithm engineering, and data science.(c)Representative state-of-the-art solutions across six domains discovered by SIMPLETES using a single open-source model as the gener- atorG, demonstrating the effect of scaling evaluation-driven discovery loops. Page 4 of110 Evaluation-driven Scaling for Scientific Discovery 72 instances spanning IBM Q20, Google Willow, and IBM Heron, with the largest gain on IBM Q20 where added CNOT overhead drops from 60,189 to 45,441, a 24.5% reduction relative to LightSABRE; on zoned neutral-atom architectures [70], it reduces geometric-mean execution time by 33.2% across 36 diverse cir- cuits, improving 34 of 36 cases with robust gains in both small- and large-scale settings. GPU Kernel Optimization.On GPU kernel optimization problems, SIMPLETES delivers strong results across both high-level operators and lower-level primitives. On TriMul (Triangle Multiplicative Update), a core operation in protein structure prediction models [1,138] that refines pairwise residue representations, our SIMPLETES-discovered Triton program attains the best performance among all compared AI methods on H100, reaching 1.122 ms, and further generalizes across hardware. In particular, it not only surpasses prior AI baselines, but also outperforms the best public GPUMode Triton submissions on other devices, including A100, where it achieves 2.135 ms versus the leaderboard best of 2.198 ms, and MI300, where it reaches 1.352 ms versus the leaderboard best of 2.657 ms. Beyond TriMul, SIMPLETES also performs strongly on lower-level primitives. It beats cub by 0.94×–2.27× (1.52× on average) on batched cumsum and outperforms the CUDA Agent [ 28] up to 2.91x. Algorithm Engineering.In scientific computing, SIMPLETES discovers a hybrid LASSO path solver that switches between LARS homotopy and coordinate descent based on the problem geometry, achieving an average2.17speedup overglmnet[ 34] and14.08oversklearn. This demonstrates that SIMPLETES can discover genuinely different algorithmic strategies that outperform decades of expert-engineered im- plementations on real-world high-dimensional datasets, rather than merely conducting parameter tun- ing. On two AtCoder Heuristic Contest problems, SIMPLETES discovers programs that surpass all human submissions and AI baselines. On AHC058, starting from scratch with no algorithmic prior, SIMPLETES discovers a multi-restart simulated annealing program achieving a score of849,325,750, a new SOTA that outscores all human submissions and all prior AI systems, with non-overlapping score distributions across 10 independent runs confirming the robustness of the gap. On AHC039, SIMPLETES also discovers a direct polygon-based simulated annealing solver, achieving an SOTA score of567,503. submissions. Mathematics Extremal Analysis.SIMPLETES discovers new SOTA constructions for the Erdős Minimum Overlap Problem and the Autocorrelation Inequalities (AC2 and AC3), outperforming both human records and all AI baselines. For the Erdős problem, by employing a coarse-to-fine optimization pipeline, SIM- PLETES found a construction achieving0.380856. This improves upon the best human result by0.186‰, surpassing the previous AI construction (0.380871by [ 142]). Furthermore, on the autocorrelation tasks, SIMPLETES discovers novel programmatic strategies to navigate complex search spaces—such as utiliz- ing FFT-based convolutions [26] with L-BFGS-B [18] refinement for AC2, and a discrete cosine transform (DCT) [ 3] parameterization for AC3, advancing the best human bounds by6.79%and0.30%, respectively. Combinatorial Construction.SIMPLETES discovers novel structures and optimization strategies. On the Sum-Difference Problem, with post-training detailed in Section2.4, it designs a novel construction featuring a long arithmetic progression backbone with sparse fringe corrections, achieving a new SOTA score of1.144887, outperforming the best human result [45] by8.03%and AlphaEvolve V2 [38] by2.05%. On Circle Packing in a Unit Square, SIMPLETES reaches SOTA results for bothn=26(2.635983)and n=32(2.939572)by evolving adaptive coarse-to-fine explorations and efficient linear programming rou- tines. Finally, on the Hadamard Maximum Determinant (Order29), it leverages inverse-guided hill climb- ing to achieve the lower-bound human record of3207 12 2 28 , surpassing ThetaEvolve [ 152] by62.3%. DataScience.In scaling law discovery [68], SIMPLETES identifies better laws, improving average extrapo- lation fitness by 352% over the best human-derived laws and by 17.8% over the previous best AI-discovered laws [ 69]. Remarkably, the SIMPLETES’s discovered law can be used to select the optimal hyperparameters in LLM pre-training [67]. In single-cell RNA sequencing [77,78,169], a biological engineering problem, SIMPLETES discovers a novel ensemble denoising algorithm that constructs and blends multiple indepen- dent denoising candidates via data-driven weighting, achieving a Tabula Muris score of 0.74 compared to the previous SOTA of 0.73 and generalizing to unseen tissue types. These gains come not from domain- specific engineering, but from the same principled approach applied uniformly across domains. Page 5 of110 Evaluation-driven Scaling for Scientific Discovery 2Test-Time Evaluation-driven Scaling 2.1Framework Recent studies on test-time scaling (TTS) improve model performance by increasing test-time computes, such as using more reasoning tokens [84], multi-turn sampling [54], or agentic workflows with tool use and search [22,25]. For scientific discovery, cycles of trial-and-error guided by external feedback are the foundation of breakthroughs. This motivates the question of Test-time Evaluation-driven Scaling (TES): whether and how the number of evaluation-driven loops can be effectively scaled up. TES is a special case of TTS with the major focus on problems where an evaluation surrogate is bothnecessaryandavailable, ruling out problems whose evaluation is purely subjective or currently intractable. The evaluation-driven loop can be formalized as follows. Given a problem instructionx 0 2 Xin the text space, we aim to discover a solutiony2Ythat maximizes a true underlying objective, often referred to as thegolden metric. In real-world scientific discovery, this golden metric is rarely directly accessible, so the discovery process relies on an explicit, queryable surrogate evaluatorV:Y !RM. For a candidate solutiony, the evaluator returnsV(y)=(r,m), wherer2Ris a scalar score or reward, andm2 M contains auxiliary feedback or metadata, such as verifier messages, error traces, or other task-specific information. With access to this evaluator, the question of TES can be stated precisely as: whether and how scaling the number of evaluation queriesNenables the discovery of a solution with a high scorer, which is expected to also perform well under the golden metric. The discovery process is controlled by a policyπequipped with a language modelG:X ! Y 1 . Specif- ically,πis initialized from a solutiony 0 , which may be a naive baseline or an existing strong solution, with(r 0 ,m 0 )=V(y 0 )being the initial score and feedback. Whenever the policy needs to, it can create a newproposalx n , which may include instructions, historical solutions and feedback, statistics from past attempts, and any other information needed byGto generate the next solutiony n . Once generated,y n is evaluated byV, and the resulting node(y n ,r n ,m n )2 YRMis returned to the policy for construct- ing future proposals. This process continues until the evaluator-query budgetNis exhausted, and the highest-scoring solution is returned. Notice that practical policies can be, and often are, asynchronous: multiple generation or evaluation jobs may be launched simultaneously, results may arrive out of order, and the policy may act before all earlier jobs have completed. This makes the space of possible policies extremely broad, ranging from simple refinement loops to complex evolutionary or agentic systems. A comprehensive review of existing designs ofπcan be found in Section 5.1. While these methods leverage evaluators in different ways, they often treat feedback as one component of a broader search procedure, rather thanexplicitlystudying the scaling effect of the evaluation-driven loop itself. TES instead asks what happens when the feedback-driven discovery loop is scaled up through more evaluator queries, rather than only scaling generation-side computation such as reasoning length, sampling count, or agent turns. Remark onV.The assumption of an evaluator surrogate does not guarantee its perfect alignment with the golden metric. Depending on the nature of the problem, the fidelity of the evaluatorVexists on a spectrum. It can be theexact metricwe care about, such as a mathematical verifier ensuring the strict correctness of a construction, or a program counting the exact number of circles packed in a given square; It can be anempirical estimationof the metric, such as timing a GPU kernel on a subset of test cases to approximate its overall execution speed. It can also be aheuristic proxy, such as a regression loss evaluated on a limited training dataset, which potentially correlates with the true generalization capability. This inherent discrepancy between the surrogate evaluator and the golden metric can lead to various forms of reward hacking [ 5,36,94] or overfitting [20,103], challenges that we systematically analyze in Section4. 2.2SIMPLETES The key question for TES is how evaluator queries should be used effectively. A naive best-of-Npolicy spends the entire budget on independently sampled candidates, failing to use evaluator feedback to guide 1 We fixGto a single LLM, as we aim to study the scaling effect of evaluation queries. Nevertheless, it can be replaced by more complex designs, including model ensembles [ 63,89], tool-using agentic workflows [8,22,25,110,113,163], and trainable test-time- adapted models [152,166,178]. Page 6 of110 Evaluation-driven Scaling for Scientific Discovery Algorithm 1SIMPLETES Require:instructionx 0 , generatorG, evaluatorV, initial solutiony 0 , parameter(C,L,K,Φ)2H 1:(r 0 ,m 0 ) V(y 0 ),S 0 f(y 0 ,r 0 ,m 0 )g 2:functionTRAJECTORY(S) 3:forℓ=1, . . . ,Ldo 4:x Φ(S) 5:GenerateKcandidatesfy k g K k=1 G(x) 6:Evaluate each candidate:(r k ,m k ) V(y k )fork2[K] 7:S S[f(y k ,r k ,m k )gwherek =arg max k r k 8:end for 9:returnS 10:end function 11:RunCindependent trajectories in parallel: S 1 , . . . ,S C TRAJECTORY(S 0 ), . . . ,TRAJECTORY(S 0 ). 12:returnarg max (y,r,m)2 S C c=1 S c r later attempts. A sequential refinement policy, on the other hand, uses feedback to improve later candi- dates, but commits the search to a single trajectory and can become trapped by early choices. Recent approaches demonstrate the power of iterative discovery systems that combine generation, evaluation, and refinement. Building on this evidence, this section asks a fundamental design question: how should evaluator queries be organized to use feedback most effectively? We introduce SIMPLETES, a simple algorithmic framework for scaling the evaluation-driven discovery loop at test time. The key idea is to organize evaluator queries through a compact design space:Cindependent trajectories provide global exploration,Lcommitted refinement steps accumulate feedback within each trajectory,Klocal candidates are evaluated before each commitment, andΦmaps the committed history into the next proposal. We present the pseudo-code of SIMPLETES in Algorithm 1, with its design space and analysis of each parameter specified below. We also discuss the theoretical insights for these parameter designs in SectionB. Definition 2.1.Given a problem instructionx 0 , the hyper-parameter space (design space) of Algorithm1 is defined as H= f (C,L,K,Φ):C,L,K2N + ,Φ: FinSet(YRM)!X g .(1) HereC,L,Kare the scaling dimensions of SIMPLETES,Φis a subroutine that creates new proposalsxbased on historical information, andFinSet()denotes the finite set of element instances. Terminology.Atrajectoryis a sequence ofLrefinement steps that starts from the initial node(y 0 ,r 0 ,m 0 ) and accumulates evaluated nodes into a setS. Each evaluated tuple(y,r,m)2Sis called anode, repre- senting a solution along with its score and metadata. Thecontext constructionmappingΦselects which historical nodes fromSto include in the next generation proposalx=Φ(S). By default, SIMPLETES allows each generation to condition on multiple historical nodes rather than only the immediate predecessor, en- abling flexible recombination of successful patterns across the trajectory. From sequential refinement to independent search.To understand the design choice of SIMPLETES, we start with a simple, straightforward evaluation-driven scaling policy:sequential refinement. Specifically, it generates a candidate, evaluates it, uses all historical feedback to generate a better one, and repeats.π seq corresponds to a special case in Definition 2.1parameterized by(1,L, 1,Φ). Yet sequential refinement has a fundamental limitation. Open-ended problems require multidimensional coverage: a high-quality solution must simultaneously satisfy multiple criteria (correctness, efficiency, generality, etc.). But refinements are path-dependent: the direction of early attempts largely determines the space of subsequent improvements. This mismatch creates a “Matthew Effect” where early progress Page 7 of110 Evaluation-driven Scaling for Scientific Discovery Autocorrelation Inequality performance ( ) 12481632 C = number of parallel search Erdos Min Overlap performance ( ) 12481632 K = local sample size Figure 2:Performance of SIMPLETES on two tasks (lower is better), autocorrelation inequalities (top) and Erdős min- imum overlap (bottom), under different global widthC(left) and local sample sizeK(right). Each curve shares the same query budgetN, with deeper color corresponding to largerN. The best configuration under the same budget is marked as “star”. We setK=1(left) andC=1(right) for simplicity. in one dimension attracts further refinement to that same dimension, starving alternatives and trapping the search around local optima. To overcome this limitation, SIMPLETES introduces a simple additional axis: global widthC. Specifically, we runCindependent trajectories in parallel, each maintaining its own history and exploring possible directions for improvement separately. By enabling independent exploration, global width increases the chance that at least one trajectory starts from sufficiently diverse early attempts, allowing later refinements to compound into qualitatively better solutions. For readers interested in a formal justification of this effect, we provide a mathematical model and the- orem in Section B. Here, we empirically demonstrate the phenomenon by applying SIMPLETES to two mathematical problems under different combinations ofCandL. As shown in Figure2(left), increasing Linitially improves performance but quickly saturates, consistent with the lock-in effect of a single refine- ment trajectory. In contrast, increasingCprovides a clear benefit by diversifying the committed histories explored under the same feedback-driven loop. This observation motivates global width as a key dimen- sion of SIMPLETES: before simply refining deeper, the policy should allocate evaluator queries to multiple independent trajectories. From single samples to local batches.The introduction ofCis due to the nature of the problems, whereas the introduction ofK, the local sample size, is due to the nature of the generator. Even when a trajectory has identified a promising direction, a single call to the generator can still produce a weak, noisy, or failed candidate. If such a candidate is immediately committed to the trajectory history, its er- rors may affect all subsequent refinements. This creates a local commitment risk at every refinement step. Within each search trajectory, SIMPLETES evaluates a batch ofKcandidates for each proposalxand adds only the highest-scoring one to the history. Under a fixed evaluator-query budget, introducing local sam- ple size changes the allocation from(C,L, 1,Φ)to approximately(C,L/K,K,Φ). The allocation betweenL andKcreates a trade-off: a largerKimproves the quality of each committed step, but it reduces the num- ber of refinement steps. a smaller local sample sizeK, on the other hand, makes the trajectory vulnerable to local generation noise that hurts future refinement. As shown in Figure 2, increasingKfrom1to a moderate value consistently improves performance, indicat- ing that local greedy selection helps prevent weak samples from entering the trajectory history. However, whenKbecomes too large and leaves only a small refinement depthL, the gains may saturate or even Page 8 of110 Evaluation-driven Scaling for Scientific Discovery reverse, as the trajectory no longer has sufficient room to accumulate feedback over time. This motivates local sample sizeKas another key dimension of SIMPLETES: evaluator queries within a trajectory should balance the number of refinement steps with the reliability of each committed step. From full history to compressed context.The discussion above focuses on how evaluator queries are allocated, which gives the three scaling dimensions(C,L,K). This allocation view abstracts away another practical question: how the feedback accumulated along a trajectory is presented to the generator. With an idealized generator that has unlimited context, perfect attention, and the ability to extract all useful patterns from prior attempts, the policy could simply expose the entire committed history to the generator. Thus,Φwould not affect the allocation analysis above. Real language models, however, are not ideal in this sense. The committed history of a trajectory can quickly become long, redundant, and noisy, containing solutions, scores, error messages, implementa- tion details, and partial insights. Query construction is therefore a compression problem: the policy must convert the trajectory-local history into a generation proposal that fits the model’s capacity while preserv- ing the feedback most useful for the next refinement step. We formalize this as a mapping Φ: FinSet(YRM)!X,(2) where each node(y,r,m)contains a committed solution, its scalar score, and auxiliary metadata. The design ofΦinvolves two sub-problems:history selection, which decides which previous nodes should be exposed to the generator, andprompt formatting, which decides how these nodes should be presented. We discuss concrete instantiations in Section2.3. Together, SIMPLETES provides a simple but effective abstraction for scaling evaluation-driven discovery loops. The design space(C,L,K,Φ)isolates the core dimensions of evaluation-driven scaling, including diverse exploration, feedback-driven refinement, and local greedy selection. This compact formulation gives a clean interface for studying scaling behavior while remaining practical across diverse scientific discovery problems. 2.3Implementation Details This subsection specifies the implementation details of SIMPLETES. Due to the independence of trajectories, we omit the trajectory indexcbelow. Each trajectory is exposed toΦonly a finite set SFinSet(YRM) of historical nodes(y,r,m). For clarity, we append necessary runtime metadata inm, including the pro- posal index for each node, the return index, and other lightweight bookkeeping information, such as which historical nodes were selected for context, how often each node has been selected, summarized fail- ure patterns, and optional reflections. This information is used in the definition ofΦand does not affect the algorithm design space. Proposal created with historical nodes.The main role ofΦis to decide which previously explored expe- riences should be presented in a new proposal that can maximize the potential result of this new solution. This is achieved by deciding which nodes inSshould be included inx=Φ(S). Our default approach is RPUCG, a graph-based extension of PUCT [ 107,118,119]. Whenever a new solution is generated from a prompt that conditions on certain historical nodes, the implementation stores parent–child links among the corresponding nodes inS. For a node indexed byi, letCh(i)denote all nodes that have included nodeiin their proposal (all nodes that are inspired by nodeiup to now). For each node, we maintain a propagated value U i =max r i ,γmax j2Ch(i) U j ! ,(3) whereγ2(0, 1]is a discount factor. The second term is ignored ifCh(i)is empty. Intuitively, an experience is valuable either because (1) it already has a high evaluator score (so it should be able to inspire a similarly Page 9 of110 Evaluation-driven Scaling for Scientific Discovery good solution), or (2) because its solution is inspiring and has led to strong descendants before (so we should reward the node). RPUCG balances the two reasons by the following formula: RPUCG(i)=U i +λρ i p 1+jSj 1+n i ,(4) whereλcontrols exploration,ρ i is the relative prior of nodeibased on its score percentile within the currentS, andn i is the number of timesihas been previously included in the context of a proposal. The first term favors nodes with strong propagated value, while the second is an exploration term that allows unexplored nodes (whosen i will be small) to be included. To reduce redundancy,Φselects his- torical nodes greedily according toRPUCG(i), while excluding the one-hop neighbors. We use multiple historical nodes for context construction, in contrast to sequential refinement and prior work [152,166], which condition on only a single node. We use RPUCG as the default selector in the main paper; alterna- tive selectors are described in Section D. Prompt construction.After selecting historical nodes,Φformats the next proposalxfor the generator G. Prompt design can substantially affect TES performance. Prior systems such as Asankhaya Sharma [6], Wang et al. [152], Yuksekgonul et al. [166] often use complex instructions with task-specific hints and expert-designed heuristics. Such hints may improve performance, but they are difficult to curate and may not transfer across tasks. In pilot experiments, removing these hints from Wang et al. [152] and using only a plain instruction caused a substantial performance drop, suggesting that prompt construction is itself an important design problem. SIMPLETES adopts a minimalist prompt strategy and moves most adaptation into the search procedure. We avoid expert-crafted hints and manually specified error patterns. Each prompt contains four types of information: 1.the plain task instructionx 0 ; 2.evaluation configurations, such as timeout and resource limits; 3.selected historical nodes fromS, including their scores, evaluator feedbackm, and concise summaries of useful observations; 4.optional automatically accumulated signals, such as repeated exceptions, timeout patterns, missing imports such asnumpy, package information fromrequirements.txt, LLM-based reflections over com- mitted winners, or validated artifacts that can serve as warm starts for constructive tasks. We ablate those design choices in section4.4. These components add little direct human prior knowledge. Instead, they are accumulated from ex- perience during TES and exposed through the same interfacex=Φ(S). Figure 25compares the task- instruction prompts used by Wang et al. [152] and SIMPLETES. Best-solution restart.One of the advantages of the SIMPLETES is its design choice of initial solutiony 0 , which can be a naive solution or an existing strong result. Therefore, a natural way to further scale up evaluation-driven loops is to start from solution obtained from previous SIMPLETES runs. We refer to this process asbest-solution restartstrategy, where after a complete run finishes, we use the best discovered solution to initialize another identical run: y 0 arg max ( y , r , m ) 2 S r. This will reset the trajectory histories and all implementation bookkeeping. Notice that, empirically, we observe that most continual best-solution restarts will saturate; e.g., the second or third restart run could never have improved on the initial solution. The saturation indicates that the underlying mechanism of best-solution restart remains underexplored, and therefore, we do not include this design in our main SIMPLETES design space. Throughout our experiments, we only run one additional best-solution restart unless otherwise specified. Page 10 of110 Evaluation-driven Scaling for Scientific Discovery Asynchronous execution.Algorithm1describes the logical search process, while the runtime executes generation and evaluation asynchronously. Each call toΦ(S)creates one logical local batch for one trajec- tory. AllKcandidates in the batch share the same proposalx, and the trajectory updates itsSonly after the batch has finished. Thus, the trajectory-level search semantics remain synchronized, even though generation and evaluation jobs may run concurrently and return out of order. The implementation uses two worker pools, one for generation and one for evaluation, connected by bounded queues. A local batch can be dispatched either as oneK-sample generation request or asK separate one-sample requests. The two modes are semantically identical because they share the same pro- posalxand are reduced by the same local best-of-Kcommit rule. The streamed mode improves overlap between generation and evaluation, while the batched mode reduces request overhead. To prevent the runtime from drifting too far ahead of the policy state, we apply trajectory-level backpres- sure. Each active trajectory may have only a bounded number of unresolved local batches. With the default setting, a trajectory cannot launch a second local batch until the previous one has resolved, recov- ering strict trajectory synchronization. Increasing this bound allows deeper pipelining. The generation and evaluation queues are also physically bounded; when they are saturated, submission blocks rather than discarding work. If no new proposal can be scheduled, the scheduler waits for either generation or evaluation progress. The runtime does not evict already admitted generation or evaluation jobs; control is exerted at admission time. Evaluation engineering.Executing thousands of untrusted LLM-generated programs requires strict iso- lation. To reduce cost, we do not rely on cloud sandboxing services such as E2B [31] or Daytona [29]; instead, each SIMPLETES run is evaluated on a single compute node. Each evaluation is executed in a fresh subprocess with process-group timeouts and memory limits, preventing infinite loops and resource leaks from affecting the broader system. For more complex tasks, we additionally enforce container-level isolation through Docker, with networking disabled and temporary directories partitioned to prevent cross-run contamination. To mitigate reward hacking, we implement independent score verification. The system does not trust metrics reported by generated code. Instead, an outer secure process independently recomputes final scores using isolated test data. This separation between generated programs and evaluation data prevents models from exploiting hardcoded outputs or overfitting to exposed test cases, improving the reliability of the reported results. 2.4Learning to Scale Evaluation-driven loop The SIMPLETES framework facilitates the generation of a massive amount of structured, high-quality TES trajectories. All these histories are a natural form of supervision on how to learn from prior experience for future improvements, analogous to how a human researcher’s experience accumulates into deeper sci- entific expertise. In this section, we propose a training paradigm that enables the model to systematically learn from trial-and-error experiences, thereby catalyzing subsequent scientific discoveries. From the perspective of reinforcement learning [ 44], TES appears deceptively simple: every proposed solution candidate receives evaluator feedback, and one might therefore train the model to optimize each step according to its immediate reward. This training paradigm has been adopted by a few existing at- tempts [ 152,166]. However, TES is not a short-horizon task in which each candidate should consistently achieve optimal scores; it is a long-horizon process in which early attempts may be valuable precisely because they expose failure modes, diversify the search, or create useful starting points for later refine- ment. A model trained only to prefer candidates with high instantaneous scores may therefore become overly conservative, missing the exploratory behaviors that attempt to jump out from the local optimum and enable later breakthroughs. Consequently, the goal of the post-training process should shift from maximizing the quality of every intermediate solution to developing an awareness of the topology of evaluation-driven scaling: to learn how evaluator feedback should shape the trajectory of future attempts. Trajectory-Level post-training.In this paper, we post-train models at the trajectory level, rather than the instance level, to align the optimization objective with the long-horizon nature of evaluation scaling. Specifically, we conceptualize the entire trajectory as a ”rollout” step, with each node–comprising a reason- Page 11 of110 Evaluation-driven Scaling for Scientific Discovery Algorithm 2Trajectory-Level Training for Test-time Evaluation-driven Scaling Require:problem setP, initial modelG, training iterationsT, number of trajectory rollout size ˆ C 1:D fg 2:fort=1toTdo 3:forP2Pdo 4:Run SIMPLETES with ˆ Ctrajectories using modelGonP, obtainfS c g ˆ C c=1 5:Add all trajectories to dataset:D D[fS c g ˆ C c=1 6:end for 7:Assign credit:W=CreditAssignment(D) 8:TrainingGwith Equation (5):G Train(G,D,W) 9:end for ing process and the proposed solution–representing an action. We deliberately bypass ALL intermediate rewards, and instead backpropagate the ultimate trajectory-level performance (i.e., the maximum score achieved across the entire history) to all constituent actions. By intentionally discarding myopic signals, this training objective becomes unbiased, ensuring that the model optimizes for the global breakthrough that aligns with the intention of scientific discovery, rather than local, potentially non-optimal, metrics. The pseudo code of our post-training process is provided in Algorithm2. We apply standard RLVR op- timization on a trajectory-level formulation that repeatedly executes the following steps: (1) performing SIMPLETES inference to sample ˆ Cindependent trajectories with the current modelG, (2) assigning scalar creditswfor each node based on trajectory-level scores, (3) optimizing the model with the following weighted objective: L=E (x, ˆ y,w)D " w jrj ∑ i=1 logπ θ ( ˆ y i jx, ˆ y <i ) # ,(5) whereDis the set of nodes obtained from the history,xis the prompt, ˆ yis the response composed of reasoning contents and the solution, andwis the assigned credit. Credit assignment.The fundamental challenge in trajectory-level post-training lies in the design of the credit assignment mechanism. In terms of the concrete design choices, one may directly use the absolute terminate score similar to REINFORCE [155], use a relative advantage similar to GRPO [112], or even sim- ply a binary reward similar to iterative rejection fine-tuning (IRFT) [143]. For simplicity, in this paper, we instantiate our training with the IRFT approach by assigningw=1to trajectories with the topR%scores for each task, andw=0to the remaining, which focuses the model’s learning capacity on high-quality trajectories. To improve data efficiency, we truncate and discard all nodes in a trajectory subsequent to the point where the maximal score first appears. This simple post-train approach turns out to be effective, as we will present in Section 4.2. We leave studies on more data-efficient approaches to future work. Engineering details.In practice, we adopt additional engineering designs to ensure robust convergence and computational efficiency. First, due to the high cost of TES sampling, we maintain a persistent replay buffer, i.e., instead of relying exclusively on on-policy rollouts that are discarded after a single training iteration, the historical nodes D are progressively accumulated and resampled, providing richer data for training. Besides, to avoid the cold-start problem, we set a larger rollout size ˆ Cin the first iteration to pop- ulate the replay buffer with a diverse set of trajectories. Second, rather than maintaining a static threshold, we dynamically decrease the top-tier ratioRas training continues. This adaptive mechanism prevents the volume of training data from scaling uncontrollably and ensures that the model is progressively trained on only the most elite trajectories as its performance improves. Page 12 of110 Evaluation-driven Scaling for Scientific Discovery 3Scientific Discovery Results We evaluate SIMPLETES on 21 scientific problems spanning six domains: quantum circuit compilation, GPU kernel optimization, algorithm engineering, mathematics extremal analysis, combinatorial construc- tion, and data science. These tasks cover domains of quantum computing, computer science, operations research, Mathematics, and biological engineering. A summary of problems and results is presented in Table1. Table 1:Summary of results across all 21 scientific problems.": higher is better;#: lower is better. Bold indicates state- of-the-art or matching state-of-the-art. Model being “Mixed” means results are obtained through the collaboration of multiple agents, each using a different closed-source frontier model. On the other hand, SIMPLETES always uses gpt-oss-120b. † : We achieved even better results on Erdős Minimum Overlap and Sum-Difference, e.g., when using our trained models. Please refer to the Section3.4.1and Section3.5.1for details. DomainProblem (Metric)Prev Best (Model)Prev Score SIMPLETES Quantum Circuit Compilation Superconducting Routing (added CNOTs#)LightSABRE [176] (Human)60,18945,441 Neutral-Atom Compilation (exec time#)ZAC [70] (Human)29187.719507.5 GPU Kernel Optimization TriMul (runtime in ms#)Zeyu Shen [41] (Human)1,1311,122 Batched Cumsum (runtime in ms#)Cub [90](Human)147104 Asymmetric Matmul (runtime in ms#)CUDA Agent [28] (Seed-1.6)747440 Algorithm Engineering Lasso Path (runtime in ms#)glmnet [34] (Human)4,1392,502 AHC039 (score")T-Discover [166] (gpt-oss-120b)567,057567,503 AHC058 (score")T-Discover [166] (gpt-oss-120b)848,305,646849,325,750 Mathematics Extremal Analysis Erdős Minimum Overlap (overlap#)TogetherAI [142] (Mixed)0.3808710.380868 † Autocorrelation Inequality 1 (bound#)TogetherAI [142] (Mixed)1.5028621.503871 Autocorrelation Inequality 2 (bound")TogetherAI [142] (Mixed)0.9612060.962694 Autocorrelation Inequality 3 (bound#)TogetherAI [142] (Mixed)1.4545551.453675 Combinatorial Construction Sum-Difference (ratio")AlphaEvolve V2 [38] (Gemini-2.0 Pro)1.1219361.143975 † Circle Packingn=26 (sum of radii")AlphaEvolve V2 [38] (Gemini-2.0 Pro)2.6359832.635983 Circle Packingn=32 (sum of radii")AlphaEvolve V2 [38] (Gemini-2.0 Pro)2.9395722.939572 Hadamard Maximum Determinant 29 (determinant")Orrick [92] (Human)0.9356730.935673 Data Science Scaling Law Discovery-parallel (R 2 ")SLDAgent [69] (GPT-5)1.0001.000 Scaling Law Discovery-domain_mix (R 2 ")SLDAgent [69] (GPT-5) 0.988 0.991 Scaling Law Discovery-lr&bsz (R 2 ")SLDAgent [69] (GPT-5)0.6040.712 Scaling Law Discovery-u_shape (R 2 ")SLDAgent [69] (GPT-5)-0.305-0.008 Single-Cell RNA-Seq Denoising (score")T-Discover [166] (gpt-oss-120b)0.730.74 SIMPLETES Settings.Unless otherwise specified, we instantiate SIMPLETES with global widthC=32, re- finement depthL=100, and local sample sizeK=16, resulting in a total evaluation budget ofN=51.2K. We use open-sourced gpt-oss-120b and gpt-oss-20b as the generatorG. We employ the vLLM inference engine as the backend forG. To explicitly constrain the token budget for a single program generation, we adopt a token-forcing strategy. The total context window size is configured to 49,152 tokens, which is strictly partitioned: a maximum of 15,536 tokens is allocated for the generated program, while the combined maximum for the input and the reasoning process is capped at 33,616 tokens. For the model generation parameters, the reasoning mode is set to “high” and the temperature is set to 1.0 . We use RPUCG as the default sampling strategy, with its hyperparameters set toλ=1.0andγ=0.8. Once the evolution has finished, we take the solution with the highest score as the eventual output. As mentioned in Section 2, this solution can be reused as the initial solutiony 0 to instantiate another brand-new evolve process. While this process can be repeated indefinitely [ 152,166], we only conduct one additional evolu- tion with the same configuration, as we empirically observe that the score will likely plateau in subsequent reruns. For all tasks considered, we report the metrics on the solution with the highest evolve score. As we have emphasized, the evaluatorVserves as a surrogate of the underlying true metrics. For each task, we include a detailed analysis of the difference between the evolve evaluatorVand the true metrics we care about. For construction tasks,Vcomputes the exact objective directly. For code optimization and data science problems,Vis an efficient and reasonable proxy, such as the performance on a subset of test cases. The entire evolution process and solution selection are based on the evaluator V instead of the true metric we report on the final solution. Please refer to our problem-specific case analysis and to Section 4.4for more details on score statistics and evolve dynamics. Page 13 of110 Evaluation-driven Scaling for Scientific Discovery Baselines.We include the best human results and recent AI-discovered results as of April 1 st , 2026, namely AlphaEvolve [38,89], OpenEvolve [6], ThetaEvolve [152], ShinkaEvolve [63], EvoX [72], T- Discover [166], TogetherAI [142], AlphaResearch [165], and KSearch [19]. As the goal is to discover better solutions, existing methods might use significantly stronger frontier models or an ensemble of them (marked as “Mixed of Models” in tables). On the other hand, all results of SIMPLETES presented in this section are obtained by open-source gpt-oss models and do not use any of the frontier closed-sourced models or fine-tuned models. Despite the gap in model capability, we show that SIMPLETES can discover SOTA solutions across most problems considered, demonstrating the power of effective evaluation scaling. For more studies on the scaling of evaluation combined with frontier models or with models taught to scale evaluation, please refer to Section2.4and Section4.4. For fair comparison, we control the task definition to be consistent with existing works, especially those settings that can influence the eventual performance. For instance, for extremal analysis and combinatorial construction, we keep the time limitation of the solutiony(which is a search program that outputs the construction of interest) identical with [6]. For algorithm engineering and kernel optimization, the final evaluation set, the precision requirement, as well as the timeout threshold are kept identical to [19,166]. Throughout the experiments, we identify severalreward hackingphenomena in which the imperfection ofVis leveraged by the solution in an irrational manner, a phenomenon also observed in prior studies. Please refer to the case analysis in each task and Section4.3for details. For SIMPLETES results, an additional manual check is conducted to ensure that the solutions are valid and reasonable. 3.1Quantum Circuit Compilation Quantum circuit compilation tasks require mapping logical quantum circuits to physical hardware while minimizing overhead from execution. The challenge lies in navigating hardware constraints while pre- serving circuit semantics. To make this challenge more precise, we briefly recall the basic principles of quantum computing from which it arises. Quantum computing is a paradigm of computation that exploits the principles of quan- tum mechanics to process information in ways fundamentally inaccessible to classical computers [ 88,97]. Whereas a classical bit occupies a definite state of 0 or 1, a quantum bit (commonly denoted as aqubit) can exist in a coherent superposition of both states simultaneously. This, combined with entanglement be- tween qubits, allows a quantum computer to represent and manipulate an exponentially large state space with a linear number of physical subsystems. Theoretically, quantum algorithms have been shown to offer substantial asymptotic speedups over classical methods for various important problem classes, including integer factoring [ 116], unstructured search [43], and simulation of quantum physical systems [33,74]. In practice, the field is now advancing from the NISQ (Noisy Intermediate-Scale Quantum) era [ 97] toward an early fault-tolerant regime [98], with the prospect of moving beyond proof-of-principle demonstrations of quantum advantage. In the circuit model, a quantum algorithm is built from one- and two-qubit gates on logical qubits. This can be expressed in a linear algebra form: the state of a qubit is a vector in a complex Hilbert space, a one- qubit gate is represented by a22unitary matrix, and a two-qubit gate by a44unitary matrix acting on two qubits. Standard two-qubit gates include the entangling controlled-NOT (CNOT) and controlled-Z (CZ) gates, as well as the SWAP gate, CNOT= 0 B B B @ 1 0 0 0 0 1 0 0 0 0 0 1 0 0 1 0 1 C C C A ,CZ= 0 B B B @ 1 0 0 0 0 1 0 0 0 0 1 0 0 0 01 1 C C C A ,SWAP= 0 B B B @ 1 0 0 0 0 0 1 0 0 1 0 0 0 0 0 1 1 C C C A . CNOT flips the target conditioned on the control, CZ applies a1phase only when both two qubits are at state 1 (and does not add a phase otherwise), and SWAP exchanges the states of two qubits. Together with single-qubit gates, CNOT or CZ can serve as a standard entangling primitive. In logical circuits, these gates can be executed between arbitrary two qubits. However, in realistic hardware, qubits are embedded in a physical architecture with locality constraints, so a two-qubit gate can only be executed when the participating qubits can directly interact. To run a logical circuit on the hardware, the compiler must therefore decide how logical qubits are assigned to physical qubits at the start of execution and how that assignment should change over time so that each two-qubit gate becomes executable. Because two- Page 14 of110 Evaluation-driven Scaling for Scientific Discovery qubit gates are among the most costly and error-sensitive operations on current quantum hardware, the overhead introduced by this compilation procedure is a central concern. Minimizing this overhead is thus a central optimization goal of quantum circuit compilation. This is a shared problem across major hardware platforms, although the underlying physical mecha- nism differs. In superconducting processors, qubits lie on a fixed sparse coupling graph, so two-qubit gates can only be applied along hardware edges and nonlocal interactions must be realized with inserted SWAPs [66]. In trapped-ion systems, a single chain can provide native all-to-all connectivity, but scalable QCCD-style architectures distribute ions across multiple zones, so ions must be shuttled into the same interaction region before a two-qubit gate can be applied [102]. In neutral-atom quantum computers, two-qubit gates are limited by finite Rydberg interaction range, so only qubits within a local interaction neighborhood can be entangled directly [70]. Although this problem is described by different terms across platforms, its core remains the same: bring the right qubits together at the right time while adding as little overhead as possible. 3.1.1Qubit Routing on Superconducting Quantum Computer Overview.On superconducting quantum computers, this compilation task is typically calledqubit rout- ing, and is often discussed together withqubit mappingorqubit allocation[27,121]. Superconducting quantum computers encode qubits in electrical circuits fabricated on a chip, typically using Josephson- junction-based devices operated at cryogenic temperatures [62]. Because the qubits are fixed on the chip, their native two-qubit interactions are constrained by a sparse coupling graph, such as a grid lattice or heavy-hex layout, whose low degree helps reduce frequency collisions and crosstalk while remaining scalable. When two logical qubits that must interact are not adjacent on this graph, routing is typically achieved by inserting SWAP gates to move their quantum states through the processor. These inserted SWAPs add extra two-qubit operations and increase circuit depth, which can substantially reduce fidelity on noise-sensitive superconducting processors. Minimizing such SWAP overhead is therefore a key objec- tive. Routed Physical Circuit (Output) Logical Circuit (Input)HardwareTopology(Input) q0 q1 q2 q3 q4 Initial layout p0 p1 p2 p3 p4 p5 SWAP q0 q3 q2 q1 q4 SWAP q0 q3 q1 q2 q4 p1 p0 p2 p3 p4 p5 q0 q1 q2 q3 q4 Figure 3:Illustration of qubit routing. A logical circuit specifies two-qubit gates between arbitrary logical qubits, while the hardware topology restricts native two-qubit interactions to edges of a coupling graph. Routing chooses an initial mapping and inserts SWAP operations to update the mapping online so each two-qubit gate becomes executable when needed. From a complexity perspective, qubit routing is closely related to token swapping and graph reconfig- uration and is NP-hard in general [55,121]. The practical challenge is amplified by long-horizon de- pendencies: a SWAP that makes one gate executable also changes the placement for all future gates, Page 15 of110 Evaluation-driven Scaling for Scientific Discovery creating a large combinatorial decision space. A substantial literature addresses this problem with a tradeoff between quality and runtime, spanning heuristic compilers such as SABRE [66] and its variant LightSABRE [176], industrial toolchains such ast|ket>[122], noise-aware routing method [85], exact optimization on small instances [86,132], and learning-based policies that search beyond hand-designed heuristics [96,120,136]. In this work, we treat routing as an algorithm-discovery target: the goal is to automatically find a stronger mapping and routing policy while preserving correctness guarantees. Problem 3.1.1 (Qubit Routing) LetC=(g 1 , . . . ,g m )be a logical circuit onnqubitsfq 0 , . . . ,q n1 g, where each two-qubit gateg i acts on a pair(a,b). The target device is a coupling graphG=(V,E)withjVj n. Aninitial mapping π 0 :fq 0 , . . . ,q n1 g!Vis an injection assigning logical qubits to physical locations; each SWAP(u,v) with(u,v)2Etransposes the logical qubits atuandv, yielding an updated mappingπ t . GivenC andG, find an initial mappingπ 0 and a sequence of SWAP insertions such that (i) every gateg i is executable under the mapping in effect at its execution time, and (i) the total number of inserted SWAPs is minimized. Q20 20 qubits, 43 couplers Willow 105 qubits, 182 couplers Heron 156 qubits, 176 couplers Figure 4:Benchmark topologies used in the routing study: the originalQ20graph from the SABRE benchmark, plus the larger GoogleWillowand IBMHeronextensions. Experimentalsetting.Our experimental scaffold fixes a complete routing engine (circuit parsing, depen- dency tracking, legality checks, and output construction) and exposes only thedecision policy. Concretely, the mutable surface contains two hooks:(i) initial layout construction(a mapping from logical to physical qubits) and(i) online SWAP selection(choosing a legal SWAP edge at each routing step). This design maximizes the algorithmic search space, allowing the model to implement a broad range of routing strate- gies. At the same time, it cleanly separates SWAP selection from gate execution, guaranteeing correctness by preventing the model from exploiting invalid gate operations. Because the instruction must expose the scaffold interfaces and is therefore longer than in other tasks, we use a 65,536-token context window, reserving 10,240 tokens for the generated program and the remaining 55,296 tokens for the input and reasoning budget. Instruction.The model is provided with the formal statement of the qubit routing problem, qualitative descriptions of the different topologies, and program interfaces of the fixed routing engine. It is instructed to maximize the combined score, with the scoring criterion given explicitly. Initial program.The initial policy is a refactor of Qiskit’s released LightSABRE Rust implementation, em- bedded inside the fixed engine. The initial layout procedure is simplified to encourage diverse exploration, while the routing policy stays effectively the same as the LightSABRE regime. This baseline provides a strong hand-engineered starting point with a well-studied quality–runtime tradeoff. Evaluation.Candidate policies are compiled and executed on a benchmark suite. Routing cost is measured by the number of inserted SWAPs, reported as added two-qubit gate cost following the convention from Li et al. [66] (i.e.1SWAP=3CNOTs). The evaluator aggregates a weighted improvement relative to the SABRE baseline for each benchmark case asS combined = ∑ i w i (s orig i s cand i ). We evaluate on the 24 circuits from the original SABRE study [ 66], across three coupling graphs (see Page 16 of110 Evaluation-driven Scaling for Scientific Discovery Figure4): IBM’s 20-qubitQ20, Google’s 105-qubitWillowprocessor [39], and a 156-qubit IBM heavy-hex- familyHeron[51,87], yielding 72 routing instances in total. Result analysis.Table2reports the performance of routing algorithms discovered by SIMPLETES. For consistency in the discussion below, we focus the detailed analysis on the algorithm discovered with gpt- oss-120b, because it has more balanced performance across the benchmark suite. The discovered router reduces the overhead of two-qubit gates substantially. Table 2shows that, across the full suite, SIMPLETES improves by 21.7% relative to SABRE and by 14.9% relative to LightSABRE when aggregated over all three topologies. Gains are largest on the small, heterogeneousQ20device (33.3% vs. SABRE, 24.5% vs. LightSABRE), while remaining consistently positive on the larger graphs (15.9% and 15.8% vs. SABRE on WillowandHeron, respectively). Figure5highlights the strongest per-circuit improvements. ModelTopology SABRELightSABRE CNOTs OursRel. W/T/L CNOTs OursRel. W/T/L gpt-oss-20b Q2068,142 42,540 37.6% 13/6/5 60,189 42,540 29.3% 11/4/9 Willow115,056 96,405 16.2% 20/1/3 110,406 96,405 12.7% 15/2/7 Heron150,558 129,096 14.3% 16/2/6 137,481 129,096 6.1% 10/3/11 gpt-oss-120b Q2068,142 45,441 33.3% 14/7/3 60,189 45,441 24.5% 10/6/8 Willow115,056 96,774 15.9% 21/1/2 110,406 96,774 12.3% 15/5/4 Heron150,558 126,822 15.8% 21/2/1 137,481 126,822 7.8% 13/2/9 Table 2:Per-topology comparison on the full benchmark suite. For LightSABRE, we choose the configuration of swap_trials=20, layout_trials=20, max_iterations=4from the original paper. sym9 9symml rd73 rd84b sqn cycle10 sym6 co14 radd z4 Quantum circuit −10 0 10 20 30 40 50 Relative improvement ratio (%) Q20 4mod5 decod24 radd rd73 sym9 z4 sqn sym6 9symml rd84b Quantum circuit −10 0 10 20 30 40 50 Willow 4mod5 decod24 sqn 9symml sym9sym6 4gt13 rd73 rd84b z4 Quantum circuit −10 0 10 20 30 40 50 Heron Top-10 relative-improvement cases vs. SABREvs. LightSABRE Figure 5:Top relative-improvement cases on each topology. Each panel shows the strongest cases by relative improve- ment, comparing against both original SABRE and LightSABRE. We omit cases where either baseline does not insert CNOTs, since the ratio would be undefined there. The learned improvements can be summarized as follows. First, the discovered algorithm invests heavily ininitial layout: it seeds high-degree logical qubits onto central, high-degree physical qubits, and then refines the mapping through an aggressive stack of hill-climbing and restart-based local search. Second, it strengthensonline SWAP selection: it broadens the candidate neighborhood beyond front-layer incident edges to include look-ahead and shortest-path edges, changes the look-ahead term into dynamic horizon with exponential decay, and reshapes the swap objective to explicitly reward immediate gate executabil- ity. Together, these changes preserve the overall LightSABRE-style structure while materially improving robustness against long-range interactions and stagnation. On already easy, low-overhead circuits (e.g., small QFT instances) where LightSABRE already inserts very few SWAPs, any more aggressive look-ahead can occasionally over-commit so the policy might overreach. Page 17 of110 Evaluation-driven Scaling for Scientific Discovery p7 p8 p9 p12 p13 p14 p17 p18 LightSABRE: 24 CX, 16 SWAP p1 p2 p6 p10 p11 p12 p16 SimpleTES: 24 CX, 0 SWAP (a)Clip ofsym9_193onQ20. p69 p70 p71 p72 p73 p78 p79 p89 p90 p91 p92 p93 LightSABRE: 20 CX, 27 SWAP p63 p64 p65 p66 p67 p68 p69 SimpleTES: 20 CX, 13 SWAP (b)Clip ofsym9_193onHeron Figure 6:Case-study clip onQ20andHeron, comparing an aligned window from LightSABRE and SIMPLETES routed circuits. We remove all single-qubit gates and show only two-qubit interactions plus explicit SWAPs. Page 18 of110 Evaluation-driven Scaling for Scientific Discovery Case study.TheQ20topology illustrates the value of shaping online SWAP decisions. Onsym9_193and 9symml_195onQ20topology, two circuits with large baseline routing overhead, SIMPLETES saves 6,810 and 7,407 relative to original SABRE, and 6,006 and 5,988 added CNOTs relative to LightSABRE. In this regime, sparse connectivity and heterogeneous node degrees create long-range two-qubit demands that a purely distance-sum heuristic might be trapped in local minima. This could be attributed to the longer dynamic lookahead horizon and scoring mechanism. Ablation study shows removing look-ahead changes reduces the improvement to 9.9% and 2.1% with respect to SABRE and LightSABRE separately. On larger graphs, the main advantage shifts from online control to initial placement: most benchmark circuits occupy a small fraction of available qubits and routing quality is determined by whether the compiler first identifies an effective compact working region. Ablation study shows this is especially effective onHeron, where improvement reduced from 15.8% to 10.8% compared to SABRE and from 7.8% to 2.3% compared to LightSABRE without this layout policy. Figure 6shows clips from routed traces ofsym9_193on both Q20andHeron, after filtering out all single-qubit gates. OnQ20, within the selected 24-CX window, the LightSABRE trace contains 16 explicit SWAPs, whereas the SIMPLETES trace contains none. OnHeron, the LightSABRE trace enters a long SWAP burst after an initially executable CX onp71andp72, requiring 27 SWAPs in total, whereas SIMPLETES completes the same window with 13 SWAPs. This example illustrates how LightSABRE can fall into locally inefficient routing cascades when its heuristic term fails to make effective progress. Experiment with Gemini.We also performed the experiment using gemini-3-pro-preview, and it deliv- ered a particularly interesting set of designs. The central idea is to make both the initial placement and the online routing policy more explicitly aware of circuit structure and hardware geometry. On the layout side, the discovered program classifies each logical interaction component according to its morphology (for example, path-like, ring-like, or dense), selects physical regions whose shape and connectivity bet- ter match that interaction pattern, prioritizes central and well-connected physical qubits for high-traffic logical qubits, and then refines the assignment through large-scale randomized restarts and local improve- ment. On the online routing side, it strengthens SWAP selection with criticality-aware scoring: both front- layer and look-ahead gates are weighted by reverse depth and critical-path relevance, while stronger decay and centrality-sensitive preferences help the search avoid stagnation and steer active qubits toward more useful regions of the device. Quantitatively, this discovered program reduces added CNOTs by 24.7% relative to SABRE and 18.3% relative to LightSABRE, with the largest gains on theQ20topology where the reductions reach 40.7% and 32.9% respectively. These learned modifications suggest that effective qubit routing benefits from a more explicitly graph-theoretic treatment, in which circuit morphology, interac- tion criticality, and hardware connectivity are exploited jointly. 3.1.2Compilation for Zoned Neutral Atom Quantum Architecture Overview.Neutral-atom platforms combine long coherence times, highly parallel native entangling op- erations, and the ability to reconfigure qubit connectivity through atom transport, which makes them a promising route toward large-scale quantum computing [13,14]. To be more precise, two-qubit gates in neutral-atom quantum computing rely on the Rydberg interaction, so the atoms participating in the same gate must be brought within the Rydberg radius before the gate can be executed. For an arbitrary logical circuit, the compiler therefore has to determine how to move the atoms involved in each required two-qubit interaction to executable physical locations [ 125,133]. In a monolithic architecture, the global Rydberg pulse used for a two-qubit layer illuminates all atoms in the single zone, including qubits that do not participate in the current layer, which introduces avoidable excitation and fidelity loss. This issue motivates zoned neutral-atom architectures, which divide the static SLM (Spatial Light Modulator) traps into a storage zone and an entangling zone: idling qubits remain in the storage zone, whereas only the qubits participating in the current two-qubit layer are moved into the entangling zone, and placed at adjacent sites where they can interact [ 14,70,125]. Atom transport is implemented by AODs (Acousto-Optic Deflectors), which consist of independently controlled horizontal and vertical tweezer lines, with each intersection between an AOD row and an AOD column creating a potential trap. Accordingly,Openactivates new AOD rows or columns, and atoms are transferred into AOD traps when these newly activated lines create new intersections that coincide with occupied atomic sites.Closedeactivates selected AOD rows or columns, and atoms currently carried Page 19 of110 Evaluation-driven Scaling for Scientific Discovery storage SLM atomentangling SLM atomAOD-carried atom AOD row/column lines ghost spot / unintended trap Initial Storage SLM Entangling SLM q0q1 q2 q3 After open Storage SLM Entangling SLM q2 q3 q0q1 After move Storage SLM Entangling SLM q2 q3 q0q1 After close Storage SLM Entangling SLM q2 q3 q0q1 Open Move Close Non-crossing Swapping the order of active AOD rows during transport is illegal. row order swapped q0 q1q0 q1 illegal order swap Ghost spots Opening two rows and two columns activates all four intersections. q0 q1 q2 intended picks: q0 and q1 q2 is unintentionally picked all active intersections become live traps Figure 7:AOD transport primitives and routing constraints in zoned neutral-atom compilation. The top row illus- trates the three basic operations.Openactivates AOD rows and columns to pick atoms from occupied SLM sites,Move translates the loaded AOD intersections to new sites, andClosereleases the transported atoms back onto the des- tination SLM. The bottom row illustrates two representative legality constraints. In the non-crossing example, two active AOD rows would need to merge into the same position or exchange their relative order, which violates the2μm minimum-spacing requirement. In the ghost-spot example, opening two rows and two columns to pick two target atoms also activates an unintended intersection, which incorrectly picks a third atom. on those lines are released back to the corresponding SLM sites.Movetranslates the currently activated AOD lines, while carrying all atoms currently loaded on the AOD. Within one rearrangement step, these operations are subject to two main shuttling constraints: Non-crossing constraint.Parallel AOD lines must remain separated by at least2μmduring shuttling, so distinct active AOD rows or columns can neither merge into the same position nor exchange their relative order [ 125,127]. Ghost-spot constraint.Activating multiple AOD rows and columns simultaneously also activates all pair- wise row-column intersections as live traps, so a legal rearrangement must avoid unintended auxil- iary traps on occupied sites [125–127]. Figure7summarizes these transport primitives and routing constraints. The top row illustrates howOpen, Move, andClosetransfer active atoms between the storage and entangling SLMs, while the bottom row shows two representative failure modes: a non-crossing violation in which two active AOD rows would have to merge or exchange order despite the2μmminimum spacing, and an unintended pickup caused by a ghost spot. Our optimization target is to minimize rearrangement time, i.e., to realize all rearrangements required for circuit execution with as few rearrangement steps as possible, with as little movement distance as possible, and therefore with as little rearrangement time as possible. EachOpenand eachClosecontributes a transfer latency ofT tran =15μs, and eachMovecontributes motion timet move = p d max /a AOD with a AOD =0.00275μm/μs 2 =2750 m/s 2 , whered max denotes the largest travel distance among the active AOD intersections [ 70]. Previous zoned compilation work has largely followed the same four-component compiler pipeline, con- sisting of ascheduler, areuse analyzer, aplacer, and arouter. Theschedulerpartitions the logical circuit into alternating one-qubit and two-qubit stages, thereby determining how many stage boundaries can trigger rearrangement. Thereuse analyzerdecides which qubits should remain in the entangling zone Page 20 of110 Evaluation-driven Scaling for Scientific Discovery across adjacent two-qubit stages, thereby reducing the need to return them to storage and reload them later. Theplacerdetermines the full spatial layout at each key stage, thereby fixing the geometric travel distances, and the opportunities for parallel atom motion. Therouterthen translates adjacent layouts into legalOpen/Move/Closesequences, thereby determining how many rearrangement steps are actually re- quired, and how much rearrangement time is accumulated. Along this line, prior work has progressed from NALAC’s abstract zoned model and logical routing formulation, to ZAC’s reuse-aware compilation, to routing-aware placement and the more scalable IDS framework [70,125–127]. Figure8shows one rep- resentativeadder_n4window under this pipeline. The left panel shows the scheduled two-qubit-stage fragment, whereas the right panels show the successive placement states and the routing groups that connect adjacent placements. Problem 3.1.2 (Compilation for Zoned Neutral-Atom Architectures) The input consists of a quantum circuit C=(Q,G,), together with the storage-site setSand the entangling-site setΩof the target architecture, whereQ is the set of logical qubits,Gis the set of logical gates, andis the partial order on the gates inG induced by gate dependencies, sog i g j means that any valid execution must executeg i beforeg j . The output includes an alternating stage division Λ=(L 1q 1 ,L 2q 1 , . . . ,L 1q m ,L 2q m ), which forms a topological layering of(G,): each gate appears in exactly one stage, gates placed in the same stage are incomparable under, and wheneverg i g j , the stage containingg i appears earlier inΛthan the stage containingg j . For each two-qubit stageL 2q t , it also includes a placement sequence, whereP t,0 is a gate placement on which the two-qubit gates inL 2q t can be executed, while P t,1 , . . . ,P t,ℓ t are zero or more auxiliary placements. Π t =(P t,0 ,P t,1 , . . . ,P t,ℓ t ), For each adjacent pair of placements in these sequences, it further includes a routing sequence that routes qubits fromP t,j toP t,j+1 , where each operationo t,j,r 2fOpen,Move,Closeg. R t,j =(o t,j,1 , . . . ,o t,j,k t,j ), The output must satisfy the following constraints.(1)Ifg i g j , theng i must be executed beforeg j . (2)At every gate placementP t,0 , the two qubits of each two-qubit gate must be within the Rydberg radius.(3)All qubits that do not participate in the current two-qubit stage must remain in the storage zone during gate execution, so as to avoid unnecessary fidelity loss.(4)All routing sequences must satisfy the AOD-related hardware constraints, including non-crossing and ghost-spot constraints. The objective is to minimize the total execution time T total =T 2q +T 1q +T move +T open +T close , whereT 2q ,T 1q ,T move ,T open , andT close denote the cumulative time spent on two-qubit gates, one-qubit gates,Moveoperations,Openoperations, andCloseoperations, respectively. Under the parameter set- ting adopted here, the durations of one-qubit and two-qubit gates are only0.625μsand0.36μs[ 133], respectively, which are much smaller than the time spent on atom rearrangement. Therefore, in this setting, minimizingT total is effectively equivalent to minimizing the rearrangement component T move +T open +T close . Unlike static routing, the main difficulty is not a single transition but the sequence of coupled layer states. A placement that shortens one rearrangement can still serialize the next once AOD constraints are enforced, and a reuse decision can save transfers in the current stage while forcing longer motion later [ 70,126,127]. Despite this long-horizon difficulty, the task is well suited to automated discovery because a candidate compiler can be represented as an executable program with fixed interfaces, and each compiled plan can Page 21 of110 Evaluation-driven Scaling for Scientific Discovery adder_n4: stages 1-3 storage qubitentangling qubitpreviousAOD batch 1AOD batch 2 q0 q1 q2 q3 S2 S1 after 2q | Reuse q2 Storage Entangling q0 q3 q1 q2 S2 before 2q | Gate (q0, q3), (q1, q2) Storage Entangling q2 q0 q1 q3 S2 after 2q | Reuse q1, q3 Storage Entangling q2q0 q1 q3 S3 before 2q | Gate (q0, q1), (q2, q3) Storage Entangling q0 q2 q1 q3 S3 S1 Figure 8:Representative compiler workflow on anadder_n4circuit window. The left panel shows a logical fragment partitioned into consecutive two-qubit stagesS1–S3. The upper-middle panel shows the placement after executing stageS1:q 3 is returned to the storage zone, whereasq 2 is reused and remains in the entangling zone for the next stage. The upper-right panel shows the gate placement for stageS2, whereq 1 ,q 0 , andq 3 are brought back into the entangling zone so that gates(q 1 ,q 2 )and(q 0 ,q 3 )can be executed. The lower-middle panel shows the placement after stageS2: q 1 andq 3 are kept in the entangling zone for reuse in the next stage, whileq 0 andq 2 are moved back to storage. The lower-right panel shows the gate placement for stageS3, whereq 0 andq 2 are reloaded into the entangling zone for gates(q 0 ,q 1 )and(q 2 ,q 3 ). In the two gate-loading transitions on the right, the required qubit rearrangement cannot be completed within a single AOD move and is therefore decomposed into two AOD batches. be checked automatically for legality and execution time. Experimental setting.Our scaffold exposes the compiler as executable code inside a fixed framework that defines the interfaces, solver, and evaluator. The compilation pipeline has four components: asched- ulerthat partitions a logicalZNAACircuitinto one-qubit and two-qubit stages, areuse analyzerthat marks qubits reused across adjacent two-qubit stages, aplacerthat outputs full placement snapshots, and arouter that converts consecutive placements into legalOpen,Move, andCloseoperations. A fixed solver invokes these components in order, and the fixed evaluator replays the emitted operations on aZNAAMachine, and rejects invalid compilations. In our setting, the editable evolve block modifies only theplacer; the sched- uler, reuse analyzer, router, solver, and evaluator remain fixed. Instruction.The instruction file includes four main pieces of information.i.An overview and problem definition together with the physical SLM/AOD model and the hard pipeline contracts, especially place- ment schema, entangling-zone adjacency, reuse semantics, and legalOpen/Move/Closebehavior.i.The compiler interfaces for the scheduler, reuse analyzer, placer, and router, together with enough fixed router code for the model to understand how routing decisions expose or serialize parallel motion.i.Guidance on interpreting evaluator feedback, including validation failures and stage-local snapshots for uneven circuits.iv.Human-written optimization insights, especially that strong placement must reason about routing concurrency over the whole placement trajectory. Initial program.The initial program exposes the editable evolve block while keeping the surrounding pipeline fixed. In Round1, that block contained a trivial placeholder placer. Each later round initialized the editable block from the strongest placer discovered in the previous round. Evaluation.We evaluate candidate placers on a 36-circuit suite built from fresh-source QASMBench [65] circuits and MQT Bench [ 101] generated circuits, spanning textbook quantum algorithms, state prepara- Page 22 of110 Evaluation-driven Scaling for Scientific Discovery tion, variational circuits, quantum simulation, quantum machine learning, and arthmetic circuits, with sizes from 6 to 500 qubits. The fixed evaluator executes each compiled plan, records correctness, total execution time, and fidelity-related losses from gate execution, transport, transfers, and decoherence, and scores each circuit against a cached baseline from the fixed scaffold. Invalid compilations receive a score of1.0. In the final round, feedback also includes compact snapshots around poorly performing two- qubit-stage windows so prompts can target uneven regressions more directly. Result analysis.Figure9reports normalized per-circuit execution times for the completed 36-circuit comparison, while Table3reports category-level and combined execution times using geometric means over circuits. The Combined row takes the geometric mean over all 36 circuits, and each category row takes the geometric mean over the circuits in that category. Table 3:Category-level geometric-mean execution time on 36 circuits. CategoryCount baseline20b-initial120b-initial120b-best-restart-3 Textbook Quantum Algorithms10 29328.1 25708.0(-12.3%) 23380.1(-20.3%) 17716.1(-39.6%) State Preparation7 16674.5 18615.0(+11.6%) 20136.6(+20.8%) 14345.3(-14.0%) Variational Circuits3 120606.8 115067.9(-4.6%) 112866.7(-6.4%) 109277.8(-9.4%) Quantum Simulation5 5853.73926.6(-32.9%)2814.8(-51.9%)2092.5(-64.3%) Quantum Machine Learning7 34286.5 43625.8(+27.2%) 35115.0(+2.4%) 26460.1(-22.8%) Arthmetic Circuits4 149012.5 194115.3(+30.3%) 156300.3(+4.9%) 111527.5(-25.2%) Combined36 29187.7 29236.2(+0.2%) 25795.5(-11.6%) 19507.5(-33.2%) qpe_n9 sat_n11 bv_n14 qft_n18qft_n29 bv_n30bv_n70 qft_n200 bv_n280 qft_n500 ghz_n23 wstate_n27 ghz_n40ghz_n78 graphstate_n60 wstate_n200 ghz_n255 vqe_uccsd_n6 0.0 0.5 1.0 1.5 Textbook Quantum AlgorithmsState PreparationVariational Circuits Circuits 1-18 gcm_n13 vqe_real_amp_n256 ising_n10ising_n26ising_n42ising_n98 ising_n420 dnn_n8 knn_n31 qugan_n39 qugan_n71 qugan_n111 dnn_n51 knn_n341 adder_n10adder_n64 multiplier_n75 adder_n433 0.0 0.5 1.0 1.5 Variational CircuitsQuantum SimulationQuantum Machine LearningArthmetic Circuits Circuits 19-36 Normalized execution time (baseline = 1) Per-circuit normalized execution time across 36 circuits baseline20b-init120b-init120b-best-restart-3 Figure 9:Normalized per-circuit execution time on the completed 36-circuit benchmark suite. The 20b-initial result is produced by running gpt-oss-20b for a single round. The 120b-initial result is produced by running gpt-oss-120b for a single round. The120b-best-restart-3result starts from 120b-initialand then continues the evolution process for three best-solution restarts. The total token budget is 65,536 for20b-initial, 49,152 for120b-initialand the first two restart rounds, and 65,536 Page 23 of110 Evaluation-driven Scaling for Scientific Discovery again in the final120b-best-restart-3round because the evolved placer code had grown too long to fit reliably under the earlier limit. Our main comparison baseline is the reuse-aware ZAC-style compiler [70]. These results can be interpreted through two comparisons. First, comparing20b-initialwith120b-initial highlights the effect of model size on the initial generated placer. Relative to baseline,20b-initialin- creases the geometric-mean execution time by0.2%and improves 12 of the 36 circuits, whereas120b-initial reaches a11.6%reduction and improves 16 circuits. Second, comparing120b-initialwith120b-best-restart-3highlights the effect of the best-solution restart process. This step improves the geometric-mean reduction from11.6%to33.2%, raises the number of improved circuits from 16 to 34, and yields the fastest execution time on 33 of the 36 circuits. The cate- gory breakdown shows that these later refinements matter not only on the already strong families, where textbook quantum algorithms improve further to39.6%and quantum simulation to64.3%, but also on the previously weaker ones. In particular, state preparation moves from+20.8%to14.0%, quantum machine learning from+2.4%to22.8%, and arthmetic circuits from+4.9%to25.2%. Variational cir- cuits remain the least sensitive category, where the final result reaches a comparatively smaller reduction ( 9.4% ). We now briefly describe the final120b-best-restart-3algorithm behind these gains. This placer replaces stage-local travel minimization with whole-trajectory routing-aware search, so it ex- plicitly optimizes the parallelism that survives under the fixed router rather than only local travel distance. It begins by constructing a moderate set of heuristic initial storage layouts, including interaction-graph- based qubit orderings, their reversed counterparts, and a few shuffled orders. From each initial layout, it runs a full forward placement pass to build a complete placement trajectory, and then compares these candidates using router-aware evaluation. Within a single forward pass, active qubits are placed stage by stage. When a stage loads qubits from storage into the entangling zone, a qubit whose partner is already reused is assigned to the complementary column of the same entangling site, whereas the remaining two-qubit pairs are assigned by Hungarian matching, with physical Euclidean distance as the placement cost. When active non-reused qubits return from the entangling zone to storage, they are grouped by source entangling row and ordered by source column. A monotone dynamic program then selects destination storage cells that preserve this order while minimizing column deviation, after which an intra-row swap refinement is applied whenever it shortens travel distance and remains compatible with the router’s non-crossing constraint. After a complete trajectory has been constructed, the algorithm evaluates the full placement sequence with the fixed router. Candidates are ranked first by the number of router-emittedMoveoperations and then, for ties, by total Euclidean travel distance. Starting from the best candidate, the algorithm performs several rounds of reverse-through-time refinement, in which a reverse pass seeded by the current final layout is fed back into a new forward pass, and then applies hill-climbing swaps on the initial storage layout. An update is accepted only when it reduces the number of router-emittedMoveoperations or preserves that count while improving travel distance. dnn_n8 q0 q1 q2 q3 q4 q5 q6 q7 bv_n14 q3 q4 q5 q6 q7 q8 q13 Figure 10:Representative scheduled two-qubit-stage fragments. The dashed barriers separate consecutive hardware stages. Thednn_n8panel shows an alternating nearest-neighbor pattern, while thebv_n14panel shows repeated interaction with a central hub qubit. Page 24 of110 Evaluation-driven Scaling for Scientific Discovery current previous batch 1 batch 2 baseline Storage Entangling q0 q1 q2 q3 q4 q5 q6 q7 120b-best-restart-3 Storage Entangling q0 q1 q2 q3 q4 q5 q6 q7 Figure 11:Representativednn_n8placement transition on one scheduled stage. A baseline-style layout spreads the active frontier across multiple entangling rows, whereas120b-best-restart-3keeps it compact on one row and re- turns qubits to storage in larger synchronized batches. Case study.Figures10and11together explain why some circuits receive only modest speedups while others improve substantially. Figure10compares the scheduled two-qubit-stage structure of the two case- study circuits, dnn_n8 and bv_n14, while Figure11shows a representative placement transition ondnn_n8. Taken together, they separate two effects: how much parallelism is already exposed by the scheduler, and how much of that parallelism the placer and router can actually preserve during rearrangement. According to figure10, the two circuits differ sharply already at the scheduler level. The bv_n14 circuit repeatedly couples different leaves to the same hub qubit, so each two-qubit stage is narrow from the outset and the room for parallel motion is limited before placement even begins. Once this schedule is fixed, later compiler stages can mainly optimize geometric factors such as approach distance and return distance, but they cannot unlock much additional concurrency. By contrast,dnn_n8alternates between wide nearest-neighbor stages on the same eight qubits, so the scheduler preserves a broad interaction frontier across consecutive stages. For this kind of circuit, execution time depends strongly on whether the placer can keep that frontier compact enough for the fixed router to realize the following transitions in parallel. Figure 11shows this second effect ondnn_n8. In the baseline-style layout, the active qubits are spread across two entangling rows, so the next rearrangement is broken into two smaller batches and more of the motion becomes serialized. The final120b-best-restart-3layout instead keeps the active frontier com- pact on one row and returns qubits to storage in larger synchronized groups, which gives the fixed router more opportunity to execute the transition with fewer sequential batches. This difference is reflected di- rectly in runtime: fordnn_n8, total time drops from 4.16 ms in the baseline to 2.70 ms in20b-initial, 2.02 ms in120b-initial, and 1.97 ms in120b-best-restart-3. Forbv_n14, where the schedule is already hub-limited, the same placement improvements have much less parallelism to exploit, so the gains are smaller and remain mostly geometric: total time decreases only from 2.78 ms in the baseline to 2.78 ms, 2.72 ms, and 2.59 ms across the three learned variants. The case study therefore illustrates the regime in which the learned placer matters most: when the scheduler leaves a wide simultaneous frontier, router- aware placement can preserve that concurrency and produce large time savings; when the circuit structure is already narrow, placement still helps, but the improvement is correspondingly smaller. 3.2GPU Kernel Optimization GPU kernel optimization tasks require generating high-performance GPU code that executes correctly and efficiently across hardware platforms. Performance depends not only on algorithmic correctness but also on low-level implementation details such as memory access patterns, thread utilization, and instruction scheduling. Experimental Setup.Our SIMPLETES supports multiple GPU programming backends, including CUDA, Triton, and TileLang. In this work, we focus on Triton to align with the implementation backend of the prior AI baselines used for comparison. Triton kernel tasks are expensive to evaluate during evolution because JIT compilation introduces substantial overhead, while reliable runtime measurement requires exclusive access to the GPU. To improve evaluation efficiency, we decompose the evaluation into acompile stage and anevalstage. Since Triton compilation is triggered on first execution rather than performed Page 25 of110 Evaluation-driven Scaling for Scientific Discovery ahead of time, we assign the initial correctness check together with first-run JIT compilation to compiler workers. Candidates that fail the correctness check under the given numerical tolerance are returned with zero reward, and only the remaining candidates are passed to eval workers for performance evaluation. GPU kernel optimization presents a substantially more complex search landscape, particularly for high- level operators such as TriMul. To enable more search iterations under a fixed evaluation budget, we reduce the local sample size to 8 and use 32 chains for all kernel optimization tasks. This design choice is supported by our ablation in results Section4.1, which show that allocating budget to more search chains is more effective than increasing the local sample size. We initialize search from the naive PyTorch implementation provided for each task. For tasks with multiple benchmark settings, we define the reward as the reciprocal of the geometric mean runtime across all settings, so that higher reward indicates better overall performance. Unless otherwise specified, all GPU-kernel experiments in this work are run on H200 GPUs. 3.2.1TriMul Overview.GPUMode [42] is an open community for GPU kernel engineering that also hosts optimiza- tion competitions on realistic operator tasks. We study our method on its TriMul (Triangle Multiplicative Update) competition [41], a core operator in AlphaFold3 [1], Protenix [138] and related protein structure prediction models. TriMul operates over pairwise representations of shape[B,N,N,C]and combines nor- malization, gated projections, and a triangle-style multiplicative interaction. The challenge lies not only in accelerating the central contraction, but also in optimizing the surrounding stages, including masking, gating, normalization, and output projection. As a result, efficient implementations must reduce inter- mediate tensor materialization and unnecessary data movement across the full operator pipeline, rather than focusing on a single inner kernel in isolation. We formulate TriMul as a high-level operator GPU kernel optimization task whose objective is to mini- mize runtime on the target GPU, subject to matching the reference implementation within the prescribed numerical tolerances. Problem 3.2.1 (TriMul) Given an input tensorx2R BNNC , a mask tensorm2f0, 1g BNN , and the operator weightsW, implement the forward pass of the TriMul operator to produce an output tensory2R BNNC . The reference operator consists of the following stages: 1.Apply input normalization toxto obtain ˆ x. 2.Compute the left, right, and output gates from ˆ x. 3.Compute the left and right projections from ˆ x, and apply masking together with the correspond- ing left/right gates. 4.Perform a triangle multiplicative interaction by aggregating over the shared index. 5.Apply output normalization, output gating, and a final projection to producey2R BNNC . Experimental setting.We evaluate the TriMul task using the official open-source GPUMode evaluation code [40], under the numerical correctness tolerance of2e-2. Because the competition had already con- cluded when we ran our experiments, we could not submit kernels to the GPUMode and therefore relied on local evaluation with the official evaluator. This evaluator provides three modes: test, benchmark, and leaderboard, which share the same task definition but differ slightly in evaluation, potentially leading to different measured performance for the same implementation. In all modes, runtime is measured using synchronized GPU event timing, with repeated runs until the relative standard error of the mean runtime falls below1e-3or the maximum number of runs (100) is reached. Since the public GPUMode rankings are based on leaderboard mode, all our main TriMul results are reported under leaderboard mode to ensure direct comparability with the published score. We compare against baselines from both public GPUMode Triton submissions and prior AI-based kernel optimization systems, including T-Discover [ 166], K-Search [19], and Aster [12]. Our SIMPLETES is run on H200, while the Triton program found is evaluated directly on NVIDIA H100, H200, A100, and AMD MI300 to assess its cross-device generalization. Page 26 of110 Evaluation-driven Scaling for Scientific Discovery Table 4:TriMul performance results on H100. Time is reported in milliseconds, and lower is better. We report the GPUMode ranking times for submissions, together with local evaluation under Triton 3.4.0 and Triton 3.6.0. The upper block lists the top-5 public GPUMode Triton submissions, excluding T-Discover. † For davidberard, we locally fix a Triton-version-related mismatch. ‡ Aster is a multi-model discovery agent. Its paper explicitly reports an 80% Gemini 2.0 Flash + 20% Claude 3.7 Sonnet configuration for the speedup analysis, but does not clearly specify a separate model mixture for the TriMul experiment. MethodModelGPUModeLocal evaluation ranking Triton 3.4.0 Triton 3.6.0 Zeyu Shen-1.1401.2930.011 1.1310.009 POLARIS AGENT -1.2951.6600.013 1.2980.009 davidberard † -1.3711.3940.011 1.3450.022 Waqar-2.3682.3490.022 2.2840.014 Arseni Ivanov-2.5462.8350.017 2.4350.003 T-Discover gpt-oss-120b w/RL 1.1611.229 0.005 1.164 0.004 AsterMulti-model agent ‡ –1.2320.007 1.2120.005 K-SearchGPT-5.2 + Gemini-3-pro–1.1690.012 1.1540.014 SIMPLETESgpt-oss-120b–1.1370.017 1.1220.008 Results analysis.Table4reports the main TriMul comparison on H100. We report results under both Triton 3.4.0 and Triton 3.6.0 because Triton version substantially affects the measured performance of Tri- ton kernels. Triton 3.4.0 is included to match the software environment used by prior AI baselines such as K-Search. However, our evaluations show that this environment does not align well with the GPUMode ranking times. For each local evaluation, we repeat the measurement three times and report the results as meanstandard deviation. Among the Triton versions we evaluated, Triton 3.6.0 yielded local evaluation results that most closely matched the GPUMode ranking times. Notably, under both Triton versions, our evolved program achieves the best performance among all compared AI methods and GPUMode Triton submissions. For K-Search, the discrepancy from their reported results (1.030 ms under Triton 3.4.0) may partly stem from differences in evaluation mode: their code uses benchmark mode, which reuses the same seed under each setting and may therefore be more sensitive to input cache behavior. For Aster, however, we are unable to verify this in the same manner, since its evaluation code is not public. Cross-Hardware Generalization.Table5further shows that the strength of our Triton kernel extends beyond the hardware used during evolution. To ensure a fair cross-hardware comparison, for all kernels from prior AI works, We directly evaluate the H100/H200-derived program on each target platform under Triton 3.6.0, without any platform-specific tuning or rerun. As a reference to the human frontier on each platform, we also report the top-3 public Triton submissions on the GPUMode for target hardware [ 41]. Across Nvidia A100, H100, H200, and AMD MI300, our SIMPLETES consistently outperforms the compared AI works and the submissions on each platform. Case Study.The performance gain of our TriMul program does not come from optimizing the core tri- angle multiplicative interaction kernel alone. Rather, it comes from reorganizing multiple operators into fewer kernels. In the initial PyTorch implementation, the TriMul operator is decomposed into a sequence of high-level operators. This decomposition incurs substantial overhead from multiple kernel launches, repeated materialization of intermediate tensors, and increased global memory traffic due to redundant loads and stores of activations. Our Triton implementation reorganizes TriMul into three main stages to reduce intermediate tensor mate- rialization, kernel launch overhead, and global memory traffic. First, we implement the input LayerNorm as a dedicated Triton kernel. Next, we fuse projection, gating, and optional masking into a single Triton kernel, which generates the left, right, and output-gate tensors together while avoiding multiple separate PyTorch operator calls and intermediate writes. For the core triangle multiplicative interaction, we retain Page 27 of110 Evaluation-driven Scaling for Scientific Discovery Table 5:Cross-hardware comparison of TriMul performance. Time is reported in milliseconds, and lower is better. † denotes the GPUMode ranking time from T-Discover. Method Time A100H100H200MI300 1st submission2.198 † 1.140-2.657 2nd submission2.3701.161 † -5.364 3rd submission4.5321.294-5.648 T-Discover2.1940.004 1.1640.004 1.0640.006 1.3820.006 Aster2.1510.004 1.2120.005 1.1010.007 1.6650.009 K-Search2.1690.013 1.1540.014 1.0750.003 1.4860.006 SIMPLETES2.1350.006 1.1220.008 1.0200.001 1.3520.004 an efficient batched matrix multiplication based ontorch.bmm. Finally, we fuse the second LayerNorm, output gating, and final projection into another Triton kernel. Scaling Evaluation-driven Loop 1.000 2.000 4.000 8.000 Time (ms) Intra-kernel refinements Kernel fusionTuning SimpleTES Discovery 8.309 5.191 3.010 2.166 1.510 1.197 1.138 1.055 1.031 1.020 FP16 compute FP32 accumulate Concat-weight single GEMM Fused LN+gate+proj Full autotune and adaptive num_warps Figure 12:The runtime of TriMul kernel during test-time evolve scaling. A second key optimization is the redesign of the precision path across the operator. In our Triton imple- mentation, activations and intermediate tensors are stored in FP16 wherever full precision is unnecessary, so as to better utilize high-throughput hardware units and reduce memory traffic.In contrast, numerically sensitive computations remain in FP32. In particular, the reductions in both LayerNorm stages are per- formed in FP32, and the final output accumulation also remains in FP32 to preserve the required output dtype. This optimization trajectory is also reflected in Figure12. The early improvements mainly come from the redesign of the precision path. Further gains then arise from kernel fusion, which removes intermediate tensor materialization and reduces kernel launch overhead across the full operator. The remaining im- provements come from hardware-specific tuning, including autotuning of warp number and stage num- ber, along with refinements to low-level kernel implementation details. 3.2.2Batched Cumsum Overview.Batched cumulative sum is a small but practically useful primitive in modern GPU work- loads. Cumsum-style computations commonly appear in sequence indexing, routing, masking, and sampling- related dataflow. Beyond LLM systems, batched cumsum is also a fundamental primitive in parallel work- loads such as stream compaction, sparse data processing, and offset generation for ragged or variable- Page 28 of110 Evaluation-driven Scaling for Scientific Discovery Table 6:Batched cumsum runtime comparison on H200. Time is reported in milliseconds, and lower is bet- ter. In the Time columns, S1–S6 correspond to(16, 32000),(16, 262208),(64, 32000),(64, 262208),(96, 201088), and (32768, 32768), respectively, where each pair denotes(bsz,n). The Score column reports the geometric mean runtime across all evaluated settings. Method Time Score S1 S2 S3 S4 S5 S6 PyTorch reference 0.063 0.430 0.068 0.465 0.375 6.496 0.357 torch.compile0.096 0.079 0.104 0.268 0.282 7.896 0.279 CUDA Agent0.104 0.082 0.110 0.173 0.182 4.749 0.228 Cub0.0330.1610.0350.173 0.143 2.215 0.147 SIMPLETES0.0350.0710.0360.081 0.079 2.186 0.104 length data layouts. Although cumulative sum is a classical primitive, its performance in batched settings still depends on how the computation is organized and mapped onto the hardware under these depen- dency constraints. We formulate batched cumsum as a data-dependent, memory-intensive GPU kernel optimization task that aims to minimize runtime on the target GPU, subject to matching the reference implementation within the prescribed numerical tolerances. Problem 3.2.2 (Batched Cumsum) Given an input tensor x 2 R BN , compute the inclusive cumulative sum along the last dimension to produce an output tensor y2R BN ,y b,i = i ∑ j=0 x b,j . Experimental Setting.We follow the KernelBench [93] Level 1 Task 89 setup for batched cumsum, and use the provided reference implementation as our initialization program. The task uses float32 inputs and the default correctness criterion with a numerical tolerance of1e-4. We measure runtime using syn- chronized GPU event timing, repeating each measurement until the relative standard error of the mean falls below1e-3or the maximum number of runs (100) is reached. To better reflect real-world workloads, we augment the original benchmark with additional shape pairs de- rived from sampling scenarios in LLM systems. The augmented shape set covers representative batch-size and vocabulary-size settings from recent models, including Mistral [ 57], GPT-OSS [91], and Gemma 3 [139], allowing us to assess whether the optimized implementation generalizes beyond the original benchmark to more realistic workloads. Our baselines include the reference implementation from KernelBench, its torch.compilevariant, CUDA Agent [28], and a library-based implementation built on cub [90]DeviceSegmentedScan. Resultsanalysis.Table6compares our program with all baselines on the KernelBench benchmark shape (32768,32768) and additional LLM sampling shape pairs. Two trends are clear from the results. First, our kernel is particularly effective under the larger-vocabulary sampling settings, where it delivers the highest speedup over the PyTorch baseline and remains clearly ahead of CUDA Agent andtorch.compile. Sec- ond, the comparison with cub reveals a workload-dependent tradeoff: Cub is slightly faster on the smaller- vocabulary cases, but our implementation overtakes it at larger vocabulary sizes, where the speedup over the high-level baselines is also most pronounced. On the large-batch KernelBench setting, our implemen- tation and cub deliver nearly identical performance, and both substantially outperform the other baselines. Case study.Our kernel uses a more specialized implementation of batched cumsum on GPU. It par- allelizes across the batch dimension, with each program handling one batch element, and further splits execution into two paths depending on the vocabulary size. In the single-tile path, when the full vocab- Page 29 of110 Evaluation-driven Scaling for Scientific Discovery ulary size (32000) fits within one tile, the kernel completes the entire cumsum locally. It first computes a local prefix sum within each thread’s vector lane, then performs a second-level scan over the resulting per-thread partial sums, and finally combines the two levels to produce the full prefix sum. In the multi- tile path, when the vocabulary size exceeds the capacity of a single tile, the kernel processes the input tile by tile while reusing the same tile-local scan structure and maintaining only a lightweight scalar running sum across tiles. This design avoids simply increasing the tile size further, since a much larger tile would raise per-program register pressure, reduce occupancy, and increase the risk of spilling data from fast on-chip storage to lower levels of the memory hierarchy, ultimately degrading performance. 3.2.3Asymmetric Matmul Overview.Asymmetric matrix multiplication is an important kernel pattern in LLM systems. Although matrix multiplication is ubiquitous, many LLM workloads involve highly asymmetric shapes, where at least one dimension is substantially smaller than the others rather than large, well-balanced matrices. Because such cases are performance-critical yet nontrivial to optimize, we include asymmetric matmul as a representative kernel task. We formulate asymmetric matrix multiplication as a compute-intensive GPU kernel optimization task whose objective is to minimize runtime on the target GPU, subject to matching the reference implementa- tion within the prescribed numerical tolerances. Problem 3.2.3 (Asymmetric Matrix Multiplication) Given input matrices A2R MK ,B2R KN , compute the matrix product C=AB,C2R MN , under a asymmetric shape in which the matrix dimensions are highly unbalanced. Experimental Setting.We follow the KernelBench Level 1 Task 9 setup for asymmetric matrix multi- plication and use the provided float32 reference implementation as our initialization program. To better reflect real-world workloads, we extend the benchmark shapes with settings derived from LLM inference scenarios. Accordingly, we use a numerical tolerance of1e-2instead of the default1e-4, so that the eval- uation more closely reflects the numerical behavior tolerated in practice. We measure runtime using syn- chronized GPU event timing, repeating each measurement until the relative standard error of the mean falls below1e-3or the maximum number of runs (100) is reached.As baselines, we compare against the reference PyTorch implementation in both FP16 and FP32, as well as CUDA Agent (TF32 variant). Resultsanalysis.Table7compares our program with all baselines on the KernelBench benchmark shape (32768, 32, 32768)together with additional asymmetric matmul cases. Overall, our SIMPLETES outperform all baselines in terms of score. A clear workload-dependent pattern also emerges from the per-setting re- sults. M shapes, S1 and S2, where the overall matrix shapes are relatively more regular, our program (w/o TF32) underperforms the CUDA Agent baseline, which is derived from the FP32 reference implementa- tion with TF32 enabled for matmul. Notably, the PyTorch FP16 baseline is also slower than CUDA Agent. This suggests that, on H200, dtype conversion introduces non-negligible overhead despite the high avail- able compute throughput. In our experiment setup, we disable the generation of any torch operators so that the discovered program remains a kernel-level implementation rather than choose the high-level li- brary calls. As a result, our kernel cannot directly incorporate the highly optimized vendor-library GEMM kernels used by CUDA Agent in some cases, even though selecting different kernel implementations for different shapes is a standard optimization strategy. To partially mitigate this limitation, we manually in- troduce a TF32 branch (w/TF32) into our kernel based on the operand shape. On the three more irregular asymmetric settings, S3-S5, our SIMPLETES consistently outperforms all baselines, achieving up to a 65.6 Case study.Our evolved implementation reorganizes asymmetric matmul around the irregularity of the workloads, rather than treating all matrix sizes with a single uniform tiling strategy. Its key design Page 30 of110 Evaluation-driven Scaling for Scientific Discovery Table 7:Asymmetric matmul runtime comparison on H200. Time is reported in milliseconds, and lower is bet- ter. In the Time columns, S1–S5 correspond to(m,k,n)settings(32, 4096, 12288),(32, 11008, 4096),(32768, 16, 32768), (32768, 32, 32768), and(32768, 64, 32768), respectively. The Score column reports the geometric mean runtime across all evaluated settings. Method Time Score S1 S2 S3 S4 S5 PyTorch FP32 (reference) 0.211 0.110 3.858 4.590 6.071 1.20 PyTorch FP160.128 0.121 3.144 3.161 3.183 0.866 CUDA Agent (TF32)0.082 0.0753.338 3.290 3.424 0.747 SIMPLETES wo/TF320.104 0.115 1.147 1.305 1.515 0.484 SIMPLETES w/TF320.082 0.0750.440 choice is to dispatch between two execution paths according to the reduction dimension K . When K is small enough to fit into one tile, the kernel loads the B tile once and reuses it across multiple row groups of A, so that the sameKNdata can be amortized over several output tiles instead of being repeatedly fetched. WhenKis large, the kernel switches to a tiled reduction with multi-stage pipelining, streaming data through the reduction dimension tile by tile. Across both paths, the kernel further adapts its tile sizes to the workload under explicit constraints from shared memory usage and register pressure. A second key ingredient is the redesign of the precision path. Starting from an FP32 PyTorch reference, our kernel transitions to mixed-precision computation when the prescribed numerical tolerance permits reduced-precision execution. Specifically, it casts the loaded A and B tiles to FP16 so as to access higher throughput tensorcore dot products. This cast is applied only when the reduction dimensionKexceeds a threshold; otherwise, the kernel directly uses the TF32 tensorcore path. One plausible explanation is that the kernel implicitly learns a tradeoff between dtype-conversion overhead and compute throughput. WhenKis sufficiently large, the cost of FP32-to-FP16 conversion can be partially amortized through over- lapped data movement and computation across multi stages, making the FP16 path advantageous. By contrast, when the stage depth is small, the additional cast overhead may become non-negligible relative to the total runtime, making direct TF32 execution preferable. However, the switching threshold in the program is set to 16, whereas all evaluated cases have reduction dimensions larger than this threshold. As a result, none of the reported experiments enter the TF32 branch. This example highlights that low- level primitives remain harder to optimize. One possible reason is that the reward signal is based only on end-to-end runtime, which provides limited information for guiding fine-grained kernel optimization. 3.3Algorithm Engineering Algorithm engineering tasks require designing efficient algorithms that perform well across diverse test cases. Success demands both algorithmic insight and careful implementation to handle edge cases and optimize for the evaluation metric. 3.3.1Lasso Regularization Path Overview.The lasso regularization path is a core computational primitive in high-dimensional statis- tics, arising naturally in cross-validation and model selection across domains from genomics to finance. Computing the full path of solutions across a grid of regularization values is orders of magnitude more efficient than solving each problem independently, but demands careful algorithmic design to exploit warm starts and sparsity structure. The de facto standard solver,glmnet[ 34], represents decades of ex- pert engineering. This task asks whether SIMPLETES can discover a solver that is faster than glmnet while maintaining the same float64 precision and correctness guarantees. Page 31 of110 Evaluation-driven Scaling for Scientific Discovery Problem 3.3.1 (Lasso Regularization Path) Given a feature matrixX2R np , responsey2R n , and a decreasing sequenceλ 1 >>λ K , produce the coefficient matrixW2R pK whosek-th column solves: ˆ w(λ k )=arg min w2R p 1 2n kyXwk 2 2 +λ k kwk 1 . A candidate solution isvalidif, for everyλ k , the objective gap relative tosklearn’s solution sat- isfiesobj( ˆ w(λ k ))obj(w sklearn (λ k ))+10 6 , checked on a held-out problem distinct from those used for timing. If any problem fails this check, the overall score is zero. The benchmark score is 1 / geomean k (solve_time_ms k )across all problem sizes if all correctness checks pass, and0other- wise. ExperimentSetting.Each candidate solution is a self-contained C++ program that reads a binary-encoded problem from stdin and writes the coefficient matrix to stdout. Correctness is evaluated on a separate fresh problem not used for timing, preventing any form of caching or test-set overfitting. The surrogate metric optimized during evolution is the geometric mean of solve times across 17 synthetic problem sizes, de- signed to cover the key axes of variation that affect lasso path solver performance: the ratio of samples to features (n/p), ranging fromnptopn; design matrix sparsity; solution density (sparse vs. dense active sets); and feature correlation, from near-independent to highly correlated Toeplitz structure. This multi-dimensional evaluation surface makes the task particularly challenging for evolutionary search. A solver that is highly optimized for one regime — say, wide problems withpn— may perform poorly on tall problems wherenp, and vice versa. If the evaluation suite is not carefully balanced across these axes, evolution can find solutions that overfit to the represented regime while generalizing poorly to others. The real-world datasets used for final evaluation (Table8) cover problems not seen during evolution, providing a meaningful test of whether the discovered algorithm generalizes. We initialize SIMPLETES with a faithful C++ port ofglmnet’s Gaussian lasso path [ 34], implementing both the covariance method (p<500) and the naive residual method (p500) with warm starts, sequential strong rule screening [141], active-set inner loop, and KKT verification. Results Analysis.Results are shown in Table8. All reported solutions pass the correctness check — the per-λobjective gap relative tosklearnis within10 6 on held-out problems for every dataset. SIM- PLETES achieves an average speedup of2.17overglmnetand14.08oversklearn. The gains are most pronounced in regimes wherenp: on the DNA dataset (1700180), SIMPLETES is9.56faster than glmnet, reflecting the advantage of a homotopy-based approach when the active set is small relative ton. On large sparse designs such as RCV1 (1720519959), SIMPLETES achieves1.76overglmnetand5.49 oversklearn. The gains againstsklearnare particularly striking on high-dimensional biological datasets —50.77on TCGA BRCA and10.16on Duke Breast Cancer. Case Study.The initialization is a faithful C++ port ofglmnet, which uses coordinate descent exclu- sively throughout — the covariance method forp<500and the naive residual method forp500— regardless of problem geometry. The best program evolved by SIMPLETES departs from this by introduc- ing a geometry-aware switching rule: for moderate-dimensional problems (p2000andnp/4), it replaces coordinate descent entirely with an exact LARS homotopy solver that traces the regularization path analytically, updating the active set at each kink with rank-1 inverse Gram updates. For wide or sparse problems, it retains coordinate descent with strong rule screening, active-set inner loop, and KKT verification. 3.3.2AtCoder Heuristic Contests Overview.AtCoder Heuristic Contests (AHC) are competitive programming competitions where par- ticipants submit programs that produce high-quality heuristic solutions to combinatorial optimization problems under strict time limits. Contests attract hundreds of participants including industry experts, making them a demanding benchmark for AI-generated algorithms. We study two tasks: AHC039 (Purse Page 32 of110 Evaluation-driven Scaling for Scientific Discovery Table 8:Lasso path solver performance on real-world datasets. All solutions pass the correctness check (per-λobjec- tive gap10 6 vs.sklearn). Times reported as mean wall-clock time (ms). Speedups are relative to SIMPLETES. Datasetglmnet (ms) sklearn (ms) SIMPLETES (ms) vs. glmnet vs. sklearn Non-biological (libsvm) Gisette (51005000)4282.74873.13141.91.361.55 RCV1 (1720519959)34521.0107790.219625.61.765.49 Biological (libsvm + TCGA/Kaggle) DNA (1700180)152.340.515.99.562.54 Leukemia (387129)20.4107.815.51.326.95 Colon Cancer (622000)15.2115.511.61.319.92 Duke Breast Cancer (447129)24.9183.718.11.3810.16 TCGA BRCA (50020238)4020.1152175.52997.21.3450.77 TCGA Liver RNA (42220168)546.43309.7374.51.468.84 TCGA Lung RNA (50020258)652.25003.6443.71.4711.28 TCGA Prostate RNA (50020232)649.913132.4438.91.4829.92 TCGA Thyroid RNA (50020164)648.17701.4442.41.4617.41 Average4139.426766.72502.32.1714.08 Seine Fishing), a computational geometry problem, and AHC058 (Apple Production Planning), a sequen- tial decision-making problem. Both have attracted strong AI systems, including ALE-Agent [ 53], ShinkaE- volve [63], and T-Discover [166]. Problem 3.3.2 (AHC039 – Purse Seine Fishing) GivenN=5000mackerel locations andN=5000sardine locations on a 2D plane, output a simple closed rectilinear polygon satisfying: at most1000vertices with integer coordinates in[0, 10 5 ] 2 , total edge length at most410 5 , all edges axis-aligned, and no self-intersections. Letabe the number of mackerel inside or on the boundary andbthe number of sardines inside or on the boundary. The per-test-case score ismax(0,ab+1). The benchmark score is the sum across all 150 test cases; a program scores zero if any test case fails a correctness check or exceeds the 2-second time limit. Problem 3.3.3 (AHC058 – Apple Production Planning) There areNL=104=40machine types in a four-level hierarchy. Each turn, the agent may spend apples to strengthen one machine or do nothing. Level-0 machines produce apples propor- tional to their count and power; higher-level machines produce lower-level machines, creating multi- plicative growth. Formally, letB i,j andP i,j denote the count and power of machinejat leveli, and let A j be the base production capacity at level 0. Each turn: apples+= N1 ∑ j=0 A j B 0,j P 0,j , (6) B i1,j +=B i,j P i,j 8i1, (7) where strengthening machine(i,j)costsC i,j (P i,j +1)apples and incrementsP i,j by 1. The per-test- case score isround(10 5 log 2 S)whereSis the final apple count afterT=500turns. The benchmark score is the sum across all 150 test cases. Experiment Setting.All programs are compiled and executed inside the ALE-Bench C++20 container [ 53]. For AHC039, following T-Discover, we initialize SIMPLETES with the same program used by ShinkaEvolve [63], derived from the ALE-Agent best solution, which would have placed 5th on the fi- nal contest leaderboard. For AHC058, we initialize SIMPLETES from scratch with a minimal program that outputs-1(do nothing) for all 500 turns, scoring zero and providing no algorithmic bias. Both tasks are Page 33 of110 Evaluation-driven Scaling for Scientific Discovery Table 9:Results on two AtCoder Heuristic Competitions. Scores reported are the maximum across 10 independent submissions to the official AtCoder platform. ALE-Agent uses Gemini-2.5 Pro for AHC039 and Gemini-3 Pro Preview high and gpt-5.2-high for AHC058. ShinkaEvolve uses an ensemble of gpt-5, gpt-5-mini, Gemini-2.5 Pro and Flash, Claude Sonnet 4, and o4-mini. MethodModelAHC039AHC058 1st human–566,997 847,674,723 2nd human–557,212 846,938,871 3rd human–554,334 846,350,877 4th human–552,933 845,489,747 5th human–549,746 845,324,831 ALE-Agent [53]Mixed550,647 848,373,282 ShinkaEvolve [63]Mixed558,026n/a T-Discover [166]gpt-oss-120b567,057 848,305,646 SIMPLETES (ours)gpt-oss-20b567,503 849,325,750 evaluated locally using the same public test case generator and seeds 0–149 as ALE-Bench [53] and T- Discover; the best evolved program is then submitted to the official AtCoder platform for final scoring. A key challenge for both tasks is score sensitivity. Because these programs operate under strict wall- clock time limits, their scores are sensitive to the execution environment — small differences in available compute determine how many iterations or search steps complete within the budget. This sensitivity directly affects the evolutionary process: when two candidate programs score similarly, the one selected as elite may reflect noise rather than genuine algorithmic superiority, potentially misleading the search direction over subsequent generations. To mitigate this, each candidate is evaluated three times across all 150 test cases per evaluation step, and the mean score is used as the candidate’s fitness for evolution. We therefore report all final scores as the maximum across 10 independent submissions to the official platform. Results Analysis.Results are shown in Table9and Figure13. On AHC058, SIMPLETES achieves a maximum score of849,325,750on the official platform, surpassing T- Discover’s maximum of848,305,646and establishing a new state of the art. The result is robust: across 10 independent submissions, SIMPLETES achieves a mean of849,030,243(σ=228,119) versus T-Discover’s mean of848,121,503(σ=119,245), with non-overlapping ranges (SIMPLETES minimum848,621,953ex- ceeds T-Discover maximum848,305,646). On our evaluation machine (AMD EPYC 9654, 96 cores, 12 workers), SIMPLETES achieves a mean of850,736,812(σ=133,794) versus850,349,796(σ=223,589) for T-Discover. On AHC039, SIMPLETES achieves a maximum of567,503, surpassing T-Discover’s maximum of567,057 and establishing a new state of the art. Across 10 independent submissions, SIMPLETES achieves a mean of566,335(σ=652) versus T-Discover’s mean of566,199(σ=669). On our evaluation machine (AMD EPYC 9654, 96 cores, 12 workers), SIMPLETES achieves a mean of569,871(σ=818) versus566,073 (σ=5,528) for T-Discover. This cross-environment sensitivity is an important practical consideration: a program that scores well on one machine may not generalize to another, which complicates both result interpretation and the evolutionary candidate selection process. Case Study.For AHC039, the best program evolved by SIMPLETES works directly on the polygon rather than on a rectangle decomposition. Initialization uses a two-stage search: a coarse200200grid applies Kadane’s algorithm to identify the highest-scoring region, then 20,000 random samples over the com- pressed coordinate space refine the starting rectangle. Simulated annealing (SA) then operates on three vertex-level moves: sliding an existing edge to a new coordinate, inserting a rectangular bulge by adding two vertices along an edge, and deleting a collinear vertex. Edge proposals are biased using two guide lists — a static list of fish coordinates1and a dynamic list of the current best solution’s coordinates — and the search restarts periodically to the best solution every 0.13 seconds. After annealing, a bidirectional Page 34 of110 Evaluation-driven Scaling for Scientific Discovery T-Discover (Previous SOTA) SimpleTES (ours) 562,000 564,000 566,000 568,000 570,000 AHC039 score 566,199 566,335 AHC039 Purse Seine Fishing T-Discover (Previous SOTA) SimpleTES (ours) 847.00M 847.50M 848.00M 848.50M 849.00M 849.50M 850.00M 850.50M AHC058 score 848.122M 849.030M AHC058 Apple Production Planning SOTASimpleTES (ours) Figure 13:Distribution of AHC039 and AHC058 scores across ten independent runs on the official AtCoder platform for SIMPLETES and Previous SOTA. On AHC058, the ranges are non-overlapping, confirming the SIMPLETES’s advan- tage is robust to run-to-run variance. On AHC039 the ranges overlap, reflecting the sensitivity of both programs to timing conditions, though SIMPLETES achieves the highest single score. greedy improvement phase tries both inward and outward one-step moves per edge and accepts any that increases the net score. For AHC058, the best program evolved by SIMPLETES operates in three phases. In the initialization phase, the program generates a diverse pool of candidate plans: one plan uses a 45-turn lookahead greedy that simulates each affordable action followed by a full greedy completion, and 13 additional restarts use ran- domized plain greedy with 30% random perturbation. The SA phase runs inside each of the 13 restarts, with each receiving an equal share of the time budget, using six mutation operators: random replacement, greedy-optimal replacement, do-nothing, swap, shift, and block move, with intermediate states cached so only the suffix from a modified turn needs recomputation. A local polish phase then greedily replaces each action with the locally optimal choice and performs adjacent-swap refinement, with any remaining time used for continued random exploration. 3.4Mathematics Extremal Analysis Mathematics extremal analysis tasks seek optimal functions that satisfy integral constraints while mini- mizing or maximizing a functional objective. These problems arise at the intersection of harmonic analysis and additive combinatorics. 3.4.1Erdős Minimum Overlap Overview.Erdős’ minimum overlap problem asks if we spread a unit amount of mass over the interval [0, 2], how should we place it so that the distribution has as little one-sided overlap as possible with the complements of its shifted copies? In the step-function surrogate used by AlphaEvolve, this becomes the search for a functionh:[0, 2]![0, 1]with unit mass that minimizes the worst translated one-sided overlap, which is sup s2[0,2] Z 2 0 h(x) 1h(x+s) dx over all shiftss[46,89,154]. The task is formulated as follows. Problem 3.4.1 (Erdős Minimum Overlap) Find a step functionh:[0, 2]![0, 1]such that Z 2 0 h(x)dx=1, Page 35 of110 Evaluation-driven Scaling for Scientific Discovery and extendhby zero outside[0, 2]. The objective is to minimize Ψ(h):=sup s2[0, 2] Z 2 0 h(x) 1h(x+s) dx. Experimental setting.Instruction.In this setting, the model is instructed to construct a discretized step function on[0, 2]that satisfies the exact discrete mass constraint ∑ h=n points /2. Optimization is restricted to the designated evolvable block, and each run is subject to an evaluation budget of1100seconds.Initial program.Following T-Discover [166], the initial script samples the discretization length, initializes the profile as the uniform functionh0.5, applies zero-mean random perturbations, and projects the result onto the feasible set defined by the box constraints and the exact mass constraint.Evaluation.The score is given by the translated-overlap objective1/Ψ(h). Table 10:Erdős minimum overlap problem results (lower values indicate better performance). The result of OpenEvolve is derived from the comparative result reported in T-Discover. The best and second-best results are highlighted inboldand underlined , respectively. Specifically, we found a better construction (0.380856) during our ablation study, as indicated by (detailed in Section4.4.1). MethodModelErdős’# Best Human [46]–0.380927 AlphaEvolve [ 89]Gemini-2.0 Pro + Flash0.380924 AlphaEvolve V2 [ 38]Gemini-2.0 Pro + Flash0.380924 OpenEvolve [6]gpt-oss-120b0.380965 T-Discover [ 166]gpt-oss-120b0.380876 Together AI [ 142]Mixed0.380871 SIMPLETESgpt-oss-20b0.380869 SIMPLETESgpt-oss-120b0.380868 Results and construction.Our best result is0.380868, marking a consistent improvement over several competitive benchmarks: it surpasses the previously reported T-Discover result (0.380876) by810 6 , and the AlphaEvolve-family result (0.380924) by5.610 5 . The gpt-oss-20b run also reaches0.380869 (a marginal difference of10 6 ), showing that the gain is mainly due to the search procedure rather than model scale alone. Interestingly, an even better result was found during our ablation study (0.380856), which is not included in the table for fair comparison. The best solution proposed by gpt-oss-120b utilizes a coarse-to-fine optimization pipeline: it first searches on a coarse discretization, then refines promising candidates with constrained local optimization and projected polishing steps. The resulting construction is a near-binary witness whose mass is arranged to flatten the overlap profile across shifts. Interestingly, by applying theBest-solution Restartstrategy (Section 2.3) to theinsp=10configuration, we achieved a new peak performance of0.380856on this task. For the sake of fairness in the comparison, we did not include it in Table 10, but we will place it in our repositories for further research. 3.4.2Autocorrelation Inequalities Overview.The First, Second, and Third Autocorrelation Inequalities (AC1, AC2, and AC3) form a com- mon family of extremal autoconvolution inequalities at the interface of harmonic analysis and additive combinatorics. For an integrable functionf:R!R, itsautoconvolutionmeasures the overlap of the function with a shifted version of itself, defined as ( f ) (t):= Z R f(tx)f(x)dx,t2 1 2 , 1 2 . All three tasks search for a candidate function (often referred to as awitnessin this mathematical context) supported on h 1 4 , 1 4 i with unit mass. They differ in their specific objectives: AC1 minimizes the peak of a non-negative autoconvolution, AC2 maximizes a norm ratio for a non-negative autoconvolution, and Page 36 of110 Evaluation-driven Scaling for Scientific Discovery AC3 minimizes the largest absolute autoconvolution value for a signed witness. The tasks are formulated as follows. Problem 3.4.2 (First Autocorrelation Inequality) Find a non-negative integrable functionf:R!R, supported on h 1 4 , 1 4 i , such that Z 1/4 1/4 f(x)dx=1. The objective is to minimize Φ 1 (f):=max t2[1/2, 1/2] (f)(t). Problem 3.4.3 (Second Autocorrelation Inequality) Find a non-negative integrable functionf:R!R, supported on h 1 4 , 1 4 i , such that Z 1/4 1/4 f(x)dx=1. The objective is to maximize Φ 2 (f):= kffk 2 2 kffk 1 kffk ∞ . Problem 3.4.4 (Third Autocorrelation Inequality) Find an integrable functionf:R!R, supported on h 1 4 , 1 4 i , such that Z 1/4 1/4 f(x)dx=1. The objective is to minimize Φ 3 (f):=max t2[1/2, 1/2] (f)(t) . Experimentalsetting.Instruction.The models are instructed to construct discretized arrays representing step functions on h 1 4 , 1 4 i that satisfy the specific non-negativity and unit-mass constraints of each task. Initial program.Following T-Discover [166], the initial scripts provide basic starting points: a linear- programming guided search for AC1, a gradient-based search for AC2, and a smooth bell-shaped envelope with oscillatory corrections for AC3.Evaluation.To cast all tasks as maximization problems, the evaluators score valid candidates by maximizing1/Φ 1 ,Φ 2 , and1/Φ 3 , respectively, though we report the standard objectives (Φ 1 ,Φ 2 ,Φ 3 ) in the main results for direct comparison. Results and construction.Our best results set new state-of-the-art records for AC2 (0.962694) and AC3 (1.453675), improving over the previous best AI results by roughly1.710 3 and2.010 3 , respectively. For AC1, our best score of1.503871remains highly competitive, though slightly trailing the leading pub- lic results. The gpt-oss-20b runs also reach strong values across all three tasks (e.g.,1.455630for AC3), showing that the performance is largely driven by the search procedure rather than model scale alone. The best programs proposed by SIMPLETES discover these solutions through distinct, simplified strategies. For AC1, the code performs a simplex-projected mass-transfer search on a1024-bin witness, concentrating most mass near the boundary. For AC2, the program uses fast Fourier transform (FFT) convolutions and L-BFGS-B refinement to produce a sparse witness characterized by a long near-flat plateau. Finally, for Page 37 of110 Evaluation-driven Scaling for Scientific Discovery Table 11:Results for Autocorrelation inequalities results. “-” indicates that the corresponding source did not report a directly aligned value. The results of ShinkaEvolve are derived from the comparative result reported in EvoX. The results of OpenEvolve are derived from the comparative result reported in EvoX and T-discover. Entries marked with † use ThetaEvolve’s corrected AC3 verifier and are therefore informative but not perfectly head-to-head with the original AlphaEvolve results. The best and second-best results are highlighted inboldand underlined, respectively. MethodModelAC1#AC2"AC3# Best human [15,82,147]–1.5097300.9015001.458100 AlphaEvolve [89]Gemini-2.0 Pro + Flash1.5053000.8962001.455700 AlphaEvolve V2 [38]Gemini-2.0 Pro + Flash1.5031700.961000- ThetaEvolve [152]Distill-Qwen3-8B1.5031330.9469001.493000 † OpenEvolve [ 6]gpt-oss-120b1.5071900.944900- OpenEvolve [6]Gemini-3.0-Pro--1.460000 T-Discover [166]gpt-oss-120b1.5028700.959100- Together AI [142]Mixed1.5028620.9612061.454555 ShinkaEvolve [63]Gemini-3.0-Pro--1.457800 EvoX [72]Gemini-3.0-Pro--1.455800 AlphaResearch [165]o4-mini--1.546000 SIMPLETESgpt-oss-20b1.5067910.9504941.455324 SIMPLETESgpt-oss-120b1.5038710.962694 1.453675 AC3, the search adopts a discrete cosine transform (DCT) parameterization to directly optimize signed cancellations, yielding the oscillatory witness. 3.5Combinatorial Construction Combinatorial construction tasks require finding discrete mathematical objects—integer sets, geometric configurations, or sign matrices—that optimize a well-defined objective. Unlike mathematics extremal analysis, these problems operate over finite, discrete search spaces. 3.5.1Sum-Difference Problem Overview.Thesums_diffsbenchmark comes from classical additive combinatorics and asks how large the normalized sumset can be relative to the normalized difference set of a single finite setAZ. The underlying quantity is closely related to the study of more-sums-than-differences (MSTD) sets [47,81]and Ruzsa-type inequalities [108,109]. Problem 3.5.1 (Sum-Difference Problem) Find a finite setAZwith at least two distinct elements. The objective is to maximize Γ(A):= log jA+Aj jAj log jAAj jAj , where A+A:=fa+a 0 :a,a 0 2Ag,A:=faa 0 :a,a 0 2Ag. Experimental setting.Instruction.The model is instructed to construct a finite integer setAof size up to512with elements bounded in[10 6 , 10 6 ].Initial program.Following AlphaEvolve V2 [ 38], the initial script starts from a fixed17-element MSTD-style seed and applies basic post-processing to deduplicate and canonicalize the set.Evaluation.The evaluator exactly computes the sumset and difference set sizes using integer arithmetic, scoring valid sets directly by maximizing the ratioΓ(A). Results and construction.Our best result sets a new state-of-the-art record of1.143975, improving over the previous best AI result (AlphaEvolve V2) by roughly0.022. Furthermore, after post-training as de- Page 38 of110 Evaluation-driven Scaling for Scientific Discovery Table 12:Sum-Difference Problem. The best and second-best results are highlighted inboldand underlined, re- spectively. Furthermore, after post-training as described in Section2.4, we further improve this score to1.144887, as indicated by (detailed in Section4.2). MethodModelsum_diff" Best human [45]–1.059793 AlphaEvolve V2 [38]Gemini-2.0 Pro + Flash1.121936 SIMPLETESgpt-oss-20b1.133720 SIMPLETESgpt-oss-120b1.143975 scribed in Section2.4, we further improve this score to1.144887, as detailed in Section4.2. The gpt-oss-20b run also reaches a strong value of1.133720, significantly exceeding the prior baseline and showing that the performance gain is largely driven by the search procedure rather than model scale alone. The best program proposed by SIMPLETES discovers these sets using an explicitly combinatorial approach: it maintains exact sum and difference multiplicity tables, alternating aggressive pruning with greedy addi- tions and local replacements. The resulting construction is highly regular. It preserves a long arithmetic progression backbone (with consecutive gaps of4) while modifying only a small number of fringe po- sitions (sparse+1,+3, and occasional+2corrections) to enlarge the sumset more efficiently than the difference set. 3.5.2Circle Packing in a Unit Square Overview.Packing unequal circles in a square is a classical problem in continuous optimization and discrete geometry [ 48,95,131]. We discuss then=26andn=32tasks together because they share the same geometric constraints and the same exact non-overlap evaluator, while differing only in instance size and search difficulty. The two problems are formulated as follows. Problem 3.5.2 (Circle Packing in a Unit Square) Find circle centers(x i ,y i )2[0, 1] 2 and radiir i 0for all1insuch that every circle lies entirely inside the unit square and no two circles overlap. Formally, r i x i 1r i ,r i y i 1r i ,81in, and (x i x j ) 2 +(y i y j ) 2 (r i +r j ) 2 ,81i<jn. The objective is max n ∑ i=1 r i . Experimental setting.Instruction.For both instances (n=26andn=32), the model is instructed to return an array of(x i ,y i ,r i )triples representing the centers and radii ofnnon-overlapping circles packed inside the unit square.Initial program.Following AlphaEvolve V2 [38], then=26andn=32initial scripts are deliberately generic. They place centers on a clipped square grid and then assign radii by greedy shrinkage subject to boundary and pairwise-distance constraints.Evaluation.The evaluator scores valid constructions by the sum of radii ∑ i r i and assigns a score of zero to invalid constructions. Results and construction.For bothn=26andn=32, our methods with different sizes of open- source models (gpt-oss-120b and gpt-oss-20b) achieve the same state-of-the-art results consistent with evolve-based scientific discovery works. Forn=26, the evolved solver adopts a coarse-to-fine strategy: broad center-set exploration followed by exact radius optimization, with the resulting packing featuring a dominant central circle, large boundary anchors, and a nearly triangular ring of medium circles; for n=32, the evolved code simplifies to a more efficient routine: it reuses the incumbent or a fixed six-row quasi-hexagonal layout, then solves the radii subproblem via linear programming, producing the more Page 39 of110 Evaluation-driven Scaling for Scientific Discovery Table 13:Circle packing in the unit square. The result of OpenEvolve is derived from the comparative result reported in CodeEvolve. ShinkaEvolve uses an ensemble of Claude Sonnet-4, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, o4-mini. The best and second-best results are highlighted inboldand underlined, respectively. MethodModelCP26"CP32" AlphaEvolve [89]Gemini-2.0 Pro+ Flash2.6358622.937944 AlphaEvolve V2 [38]Gemini-2.0 Pro+ Flash2.6359832.939572 ShinkaEvolve [63]Mixed2.635982- ThetaEvolve [152]Distill-Qwen3-8B2.635983- T-Discover [166]Qwen3-8B2.6359832.939572 CodeEvolve [7]Qwen3-Coder-30B2.6359802.939560 OpenEvolve [6]Qwen3-Coder-30B-2.931560 SIMPLETESgpt-oss-20b2.6359832.939572 SIMPLETESgpt-oss-120b2.6359832.939572 homogeneous packing. 3.5.3Hadamard Maximum Determinant Overview.The Hadamard Maximum Determinant, Order 29 task belongs to the classical maximal- determinant problem forf1, 1g-matrices, a central benchmark in extremal matrix theory andD-optimal design. Order 29 is particularly notable because it lies outside the Hadamard regime, meaning the best known constructions are near-extremal rather than exact Hadamard matrices. The task is formulated as follows. Problem 3.5.3 (Hadamard Maximum Determinant, Order29) Find a matrix A=(a ij )2f1, 1g 2929 that maximizes Λ(A):=jdetAj. Experimental setting.Instruction.The models are instructed to construct a 29 by 29 sign matrix with entries inf1, 1gthat maximizes the absolute determinant.Initial program.Following ThetaEvolve [152], the initial script begins from a quadratic-residue construction modulo 29 and applies single-entry hill climbing with simulated annealing.Evaluation.The evaluator directly computes the exact determinant using the Bareiss algorithm, reporting both the raw absolute determinantΛ(A)and the normalized score jdet(A)j/(3427 12 2 28 ). Table 14:The Hadamard Maximum Determinant, Order 29. The best and second-best results are highlighted inbold and underlined, respectively. MethodModelHM29 score"HM29 det" Best human [92]–0.9356733207 12 2 28 ThetaEvolve [152]ProRL-1.5B-v20.563500– ThetaEvolve [152]Distill-Qwen3-8B0.576400– SIMPLETESgpt-oss-20b0.9356733207 12 2 28 SIMPLETESgpt-oss-120b0.9356733207 12 2 28 Resultsandconstruction.Our best results exactly recover the long-standing classical lower-bound record of3207 12 2 28 , substantially outperforming the previously published AI baselines such as ThetaEvolve. Both the gpt-oss-120b and gpt-oss-20b runs successfully reach this optimal value, demonstrating that the performance is driven by the search procedure rather than model scale alone. Page 40 of110 Evaluation-driven Scaling for Scientific Discovery The solver proposed by SIMPLETES achieves this by broadening the initializer into a diverse seed pool— including incumbent, quadratic-residue circulant, orthogonal-sign, and cropped Sylvester matrices—and alternating inverse-guided multi-flip hill climbing with simulated annealing. The resulting high-determinant family is characterized by a near-orthogonal Gram structure. 3.6Data Science Data science tasks require discovering models or transformations that generalize beyond the training dis- tribution. Evaluation measures held-out performance, and overfitting is a primary concern. 3.6.1Scaling Law Discovery Overview.Scaling law discovery [68,69] studies how machine learning performance changes with scale and aims to identify compact symbolic laws that extrapolate from small-scale experiments to larger regimes. In foundation-model development, such laws are used to predict quantities such as training loss, downstream error, or task-specific metrics from variables including model size, dataset size, vocabulary size, learning rate, batch size, and architectural choices. The central challenge is not merely to fit a curve, but to recover a concise analytic form whose structure is not knowna prioriand that generalizes across related experimental settings. Each trial consists of a set of input variables, a target quantity, and a control index identifying the experimental context, such as a model family, dataset, or domain. The control in- dex makes it possible to search for a shared symbolic form across settings while allowing the coefficients to vary within each setting. This makes scaling law discovery especially suitable for SIMPLETES: candi- date laws can be represented as executable programs, evaluated automatically on held-out extrapolation performance, and compared across a large, structured design space involving symbolic form, asymptotic behavior, parameter sharing, and fitting procedures. Compared with standard regression, the empha- sis here is on symbolic structure and extrapolation; compared with broader agentic ML benchmarks, the objective is not to engineer an end-to-end workflow, but to discover a compact law that transfers across settings and remains accurate in unseen scaling regimes. We formalize scaling law discovery as follows. Problem 3.6.1 (Scaling Law Discovery) Let D train =f(x i ,j i ,y i )g m i=1 be a collection of observed trials, where eachx i 2R n is a vector of feature variables,j i 2Cis a control index denoting the experimental setting, andy i 2R k is the target quantity to be predicted. For each settingj2C, let D (j) train =f(x i ,y i ):j i =jg denote the subset of trials belonging to that setting. The goal is to discover: 1.a symbolic lawf θ :R n !R k parameterized by coefficientsθ, and 2.a fitting procedure that produces, for each control settingj2C, a parameter vectorθ j fromD (j) train , such that the instantiated predictorsf θ j extrapolate accurately to unseen inputs drawn from larger- scale or otherwise held-out regions of the input space. In benchmark form, this objective is evaluated on hidden extrapolation setsfD (j) test g j2C by maximizing the average coefficient of determination 1 jCj ∑ j2C R 2 fy:(x,y)2D (j) test g,f θ j (x):(x,y)2D (j) test g , or, equivalently, by minimizing extrapolation error on the unseen test points. A canonical example is pretraining scaling, where the inputs are model sizeNand dataset sizeD, the target is training lossL, and the objective is to discover a law of the formLf θ (N,D). The same framework also covers scaling with domain mixture, learning rate, batch size, and related variables. What distinguishes this task from ordinary regression is that success is measured primarily by extrapolation beyond the fitted regime, not by interpolation within it. Page 41 of110 Evaluation-driven Scaling for Scientific Discovery Table 15:SLDBench results. Scores are test-setR 2 values, where higher is better, averaged over 5 random seeds. AgentModelparallel domain_mix lr&bsz u_shape Avg.R 2 AiderGPT-50.9910.514-0.659-0.4740.093 Terminus-2GPT-51.0000.502-0.754-0.6040.036 Mini-SWE-AgentGPT-50.9970.873-0.269-0.4910.277 OpenCodeGPT-51.0000.960-0.368-0.4800.278 OpenHandsGPT-51.0000.899-0.909-0.2780.178 CodeXGPT-50.9990.933-0.039-0.7400.288 GooseGPT-51.0000.9440.280-0.2320.498 SLDAgentGPT-51.0000.9880.604-0.3050.572 Human–1.0000.671-0.076-1.0000.149 Gemini-CLIGemini-2.5-Flash0.2000.530-0.873-0.794-0.234 SLDAgentGemini-2.5-Flash1.0000.991-0.871-0.7580.090 Gemini-CLIGemini-3-Pro-Preview 0.6000.978-0.332-0.8470.100 SLDAgentGemini-3-Pro-Preview1.0000.9840.513-1.0000.374 Claude CodeClaude-Haiku-4.51.0000.905-0.511-1.0000.099 SLDAgentClaude-Haiku-4.51.0000.980-0.657-0.7540.142 Claude CodeClaude-Sonnet-4.50.9980.971-0.846-1.0000.031 SLDAgentClaude-Sonnet-4.51.0000.985-0.514-0.5220.237 CodeXo4-mini1.0000.553-0.773-1.000-0.055 SLDAgento4-mini1.0000.9890.611-1.0000.400 SIMPLETESgpt-oss-120b1.0000.9910.712 -0.008 0.674 Experimental setting.We report results on the four-task SLDBench subset:parallel(36 seen / 12 un- seen) [24],domain_mix(80 seen / 24 unseen) [164],lr&bsz(2,702 seen / 117 unseen) [67], andu_shape (389 seen / 127 unseen) [156]. Following the original SLDBench protocol, the unseen split is always con- structed as an extrapolation test set by holding out the largest model sizes, compute regimes, or other extreme settings, rather than by using random interpolation-style splits. The execution environment also follows SLDBench: agents operate in a sandbox terminal with a minimal Python stack (scikit-learn, pandas, anddatasets) and no network access, and must implement the discovered law and its parameter- fitting subroutine under the required function signature. Final performance is measured by test-setR 2 , clipped to[1, 1], on the hidden extrapolation split, where higher is better. We setC=16,L=20, and K=16for SIMPLETES on this task. To ensure a controlled comparison, our method uses the same initialization, task instruction, and evalua- tor as SLDAgent. In particular, the initial program is the same baseline program pair used by SLDAgent: a naive power-lawscaling_law_functogether with a standard BFGS-basedfit_scaling_lawoptimizer. We also adopt the same task-specific instruction template, which asks the model to evolve both the symbolic expression and the fitting routine from this baseline, while emphasizing extrapolation accuracy, cross- setting generalization, parameter efficiency, and numerical/theoretical stability; the instruction addition- ally provides task context, function signatures, and data characteristics such as feature definitions and value ranges. The evaluator is likewise kept identical to the SLDAgent setting. This alignment is important because SLDAgent is itself an evolution-based method built on top of the OpenEvolve framework, using iterative mutation and evaluation of candidate programs; accordingly, our comparison isolates the effect of the evolutionary strategy rather than differences in initialization, prompting, or evaluation pipeline. Results analysis.Table15shows that scaling law discovery remains a strong discriminator of agent ca- pabilities even on this reduced four-task subset. SIMPLETES achieves the best overall average score of 0.674, outperforming the strongest baseline, SLDAGENT with GPT-5, which attains 0.572 on the same subset. The tasks also reveal a useful spectrum of difficulty.parallelis close to saturated, with many methods reach- ing near-perfect extrapolation, whiledomain_mixis broadly tractable but still differentiates top-performing methods at the margin. By contrast,lr&bszand especiallyu_shaperemain challenging: many agents still obtain negative testR 2 , indicating that symbolic forms that interpolate well in-range often fail to extrapo- late reliably into the held-out regime. SIMPLETES’s advantage is most pronounced on these harder extrap- olative settings, where it attains the best score onlr&bszand the best score onu_shape, while also tying for best onparallelanddomain_mix. Page 42 of110 Evaluation-driven Scaling for Scientific Discovery (SLDAgent law) (SimpleTES law) (Optimal loss) Figure 14:Ground-truth validation-loss heatmap on thelr&bsztask for a1B-parameter LLM trained on100B tokens. Each cell reports the realized validation loss for one candidate(lr, bsz)pair on the evaluation grid. The red star marks the true best grid point,(1.38110 3 , 512), with loss2.0762. The blue star marks the point selected by SIMPLETES, ob- tained by evaluating the discovered loss law at every admissible grid point and choosing the point with the smallest predicted loss; this yields(1.95310 3 , 512), whose realized loss is2.0774. The gray star marks the SLDAgent rec- ommendation,(1.95310 3 , 384), whose realized loss is2.0776. Both methods identify the same low-loss basin, but SIMPLETES lands slightly closer to the true optimum in this extrapolated regime. This case illustrates that a symbolic law that accurately models the full loss surface can be used directly for hyperparameter selection at scales beyond the observed training runs. Case study.Thelr&bsztask is a particularly informative case study because it tests whether a discovered scaling law can support an actual decision, rather than merely fitting observed losses. The goal is to predict validation loss as a function of learning rate, batch size, dataset size, and model size, and then use the predicted surface to select a good hyperparameter configuration in an extrapolated regime. Because the held-out split consists of larger-scale configurations, success requires recovering the geometry of the loss basin beyond the observed range. Our method discovers an explicit symbolic law for the full loss surface, ˆ L SIMPLETES (lr, bsz,D,N)=6.567185 lr 0.0131 bsz 0.0096 D 0.0346 N 0.1173 +0.408323 lr 0.2807 bsz 0.4838 D 0.0499 N 0.0995 +26.840067 lr 0.0657 bsz 0.0595 D 0.2274 N 0.0474 +66071022.857 lr 0.0844 bsz 1.8723 D 1.4189 N 0.0982 +11.145293 lr 0.9783 bsz 0.1254 D 0.4142 N 0.4786 0.1294720520. Given a target regime(D,N), we evaluate this law on the full admissible hyperparameter gridGand choose the point with the minimum predicted loss: (lr † , bsz † )=arg min (lr,bsz)2G ˆ L SIMPLETES (lr, bsz,D,N). This matches the actual deployment setting: we score every feasible grid point directly, rather than opti- mizing in a continuous space and then rounding back to the nearest grid point. For the extrapolation example in Figure 14, corresponding to a1B-parameter model trained on100B to- kens, this procedure selects the blue-star configuration(1.95310 3 , 512). Its realized validation loss is2.0774, while the true best grid point is the red-star configuration(1.38110 3 , 512)with loss2.0762. Page 43 of110 Evaluation-driven Scaling for Scientific Discovery Thus, the configuration selected by SIMPLETES is only0.058%above the optimum. For comparison, SLDA- gent selects the gray-star point(1.95310 3 , 384), whose realized loss is2.0776, or0.067%above the optimum. Although the numerical gap is small, SIMPLETES more accurately identifies the optimal region of the extrapolated loss surface and lands slightly closer to the empirical optimum. This comparison is meaningful because the three marked points all lie in the same narrow low-loss valley, so the main challenge is fine-grained recovery of the learning-rate/batch-size trade-off rather than coarse localization. Modeling the full loss surface, rather than only the optimum coordinates, provides richer supervision and makes the discovered law directly usable as a practical selector in unseen regimes. More broadly, this example shows why high held-outR 2 matters in practice: the value of a scaling law lies not only in predictive accuracy, but also in its ability to support extrapolative decisions. On thelr&bsz task, SIMPLETES turns a symbolic loss law into a grid-level hyperparameter rule and selects a point that is essentially optimal on the true evaluation landscape. 3.6.2Single-Cell RNA-Seq Denoising Overview.Single-cell RNA sequencing (scRNA-seq) resolves gene expression at the level of individ- ual cells, enabling the characterization of cell types, transcriptional states, and developmental trajectories [76,78,174]. Despite recent advances in throughput, scRNA-seq measurements are subject to substantial technical noise from low mRNA capture efficiency and stochastic dropout [145], motivating dedicated denoising algorithms [146]. Denoising quality is assessed using the molecular cross-validation frame- work of Batson et al. [10], which partitions observed UMI counts into training and test sets via binomial subsampling, providing a proxy for accuracy without external ground truth. This protocol underpins the OpenProblems benchmarking suite [ 77], which provides three datasets of increasing size: Pancreas, PBMC, and Tabula Muris Senis Lung. MAGIC [146] and ALRA [71] are the strongest published baselines, with T-Discover [ 166] representing the current state of the art. The core difficulty is the tension between two complementary objectives: MSE in log-normalized space re- wards faithful recovery of relative expression levels, while Poisson negative log-likelihood rewards count- data consistency after library-size rescaling. Aggressively optimizing one tends to degrade the other, so the benchmark enforces a hard Poisson constraint to keep both in check. This task is a natural target for SIMPLETES: candidate algorithms can be expressed as executable programs, evaluated automatically on a fixed dataset split, and strong performance requires non-obvious algorithmic choices that are difficult to tune analytically. Experiment Setting.The molecular cross-validation framework [10] splits the Pancreas dataset into a training matrixX tr and a held-out test matrixX te via binomial subsampling. The algorithm observes only X tr and is evaluated on two complementary metrics: MSE in log-normalized space and Poisson negative log-likelihood, both normalized against a no-denoising baseline and a perfect-denoising oracle. The Pois- son score acts as a hard constraint — solutions that fail it are rejected — and the final score is the mean of the two normalized metrics. Each candidate program is subject to a 300-second wall-clock time limit. Following Yuksekgonul et al. [ 166], we initialize SIMPLETES with MAGIC [146]. Problem 3.6.2 (Single-Cell RNA-Seq Denoising) GivenX tr 2Z CG 0 (the training split of the Pancreas dataset), find a functiondenoise :Z CG 0 ! R CG 0 producing ˆ X=denoise(X tr )with non-negative finite entries, ˆ X max kX tr k 1 , and Pois_norm( ˆ X)0.97, all within a 400-second time limit. The benchmark score is score( ˆ X)= ( MSE_norm( ˆ X),if ˆ Xis valid, 0,otherwise. Generalization is assessed on the held-out PBMC and Tabula Muris Senis Lung datasets without re- training. Results Analysis.Table16reports generalization performance on the held-out PBMC and Tabula Muris Senis Lung datasets. Following the same protocol as Yuksekgonul et al. [166], evolution runs exclusively Page 44 of110 Evaluation-driven Scaling for Scientific Discovery on the Pancreas dataset; PBMC and Tabula are completely withheld during evolution and used only for final evaluation, making the reported scores a direct measure of out-of-distribution generalization. SIM- PLETES achieves a Tabula score of0.74, surpassing T-Discover (0.73) and matching it on PBMC (0.71), while outperforming all other baselines on both datasets. PBMCTabula MethodScore"MSE#Poisson#Score"MSE#Poisson# MAGIC0.420.190.160.400.180.12 MAGIC (A)0.420.190.160.400.180.12 MAGIC (R)0.640.190.050.640.180.03 MAGIC (A, R)0.640.190.050.640.180.03 ALRA (S, RN)0.500.260.050.470.270.03 Best-of-256000.620.200.050.650.180.03 OpenEvolve0.700.160.050.710.150.03 T-Discover0.710.150.050.730.140.03 SIMPLETES (ours)0.710.150.050.740.130.03 Table16:Denoising results on held-out PBMC and Tabula Muris Senis Lung datasets. Score is the mean of normalized MSE and Poisson scores (higher is better). MAGIC (A) = MAGIC approximate; MAGIC (R) = MAGIC with reversed normalization; MAGIC (A, R) = MAGIC approximate with reversed normalization. ALRA (S, RN) = ALRA with square-root norm and reversed normalization. All non-SIMPLETES results are taken from Yuksekgonul et al. [166]. Case Study.The init program for this task is MAGIC with reversed normalization [146], which builds a single diffusion operator on square-root-transformed, library-size-normalized counts and applies it itera- tively. T-Discover [ 166] extends this template with gene-adaptive variance-stabilizing transform ensem- bling (Anscombe, Freeman-Tukey, and square-root transforms weighted by per-gene dropout), low-rank SVD refinement, and a log-space polishing step that targets the evaluation metric directly. The algorithm discovered by SIMPLETES takes a structurally different approach. Rather than elaborating a single diffusion pipeline, it constructs multiple independent denoising candidates — including weighted multi-scale diffusion under both correlation- and Euclidean-based graph operators, log-space diffusion, raw neighbor averaging, PCA imputation, and NMF-based reconstruction — and scores each candidate internally onX tr , with no access toX te at any point. The final output is a data-driven ensemble: candi- dates are blended using weights derived from their inverse Poisson and inverse MSE losses, subject to the Poisson hard constraint, with the blending exponent selected to minimize validation MSE. A final gene- mean calibration step clips per-gene scaling factors to a narrow range around1.0, preserving count-data fidelity while reducing over-smoothing. Crucially, none of these design choices are dataset-specific — the algorithm adapts its ensemble weights entirely fromX tr at runtime, which explains its ability to generalize to the unseen PBMC and Tabula distributions, surpassing T-Discover on Tabula (0.74vs.0.73) while matching it on PBMC. Page 45 of110 Evaluation-driven Scaling for Scientific Discovery 4Method Analysis In this section, we present a comprehensive analysis of our proposed framework. We first investigate its scaling behaviors, demonstrating how systematic expansion of the evaluator-query budget drives contin- uous performance improvements. Next, we examine the impacts of training, highlighting its ability to enhance both in-domain and out-of-domain discovery capabilities. We then analyze various reward hack- ing phenomena that emerge when models autonomously exploit vulnerabilities in surrogate evaluators. Finally, we conduct detailed ablation studies to evaluate the individual contributions of core framework components. 4.1Experiments on the Scaling Behavior of SIMPLETES To investigate the scaling behavior of SIMPLETES, we explore the impact of key framework parameters: total evaluation budgetN, refinement depthL, global widthC, and local sample sizeK. We aim to reveal how the framework effectively translates increased computational scale into substantial performance improve- ments. In our ablation studies, we select three representative open-ended tasks—the first autocorrelation inequality, the Erdős minimum overlap problem, and the TriMul task in GPU kernel optimization—to investigate the scalability of SIMPLETES. Scalability of global width (C) and refinement depth (L).First, we scale along the dimensions of global parallel exploration (C) and sequential refinement depth (L). For these experiments, we hold the local sample size constant atK=32. The results are visualized in Figure15. As demonstrated in the heatmaps, SIMPLETES exhibits strong and consistent scalability along both the global width (C) and sequential refinement depth (L) axes. The consistent performance improvements observed asLincreases serve as compelling evidence for the efficacy ofsequential refinement, demonstrat- ing that iteratively leveraging historical feedback successfully drives targeted enhancements. However, because sequential refinement is inherently path-dependent and prone to saturation around local optima, scalingCprovides a critical and complementary advantage. The robust gains along theCaxis validate the power of parallel exploration: by distributing the evaluation budget across independent trajectories, the framework successfully diversifies the committed histories, effectively mitigating path-dependent bot- tlenecks and ensuring that later refinements compound upon structurally advantageous foundations. Furthermore, the heatmaps reveal that the performance benefits derived from scalingCversus scalingL are strongly task-dependent. For mathematical discovery tasks like AC1 and Erdős, scaling the number of parallel chainsC(e.g., up toC=32) provides a more pronounced advantage at larger scales. Mathemat- ical constructions often require extensive exploration of diverse starting points to discover a promising structural “flash of insight.” Conversely, for the TriMul GPU kernel optimization task, scaling the itera- tion depthLdrives the most significant performance gains. Kernel optimization heavily relies on complex, step-by-step engineering refinement rather than sudden structural breakthroughs, making deep trajecto- ries (largeL) essential to systematically tune hardware-specific parameters and iteratively squeeze out peak performance. Scalability of local sample size (K) and refinement depth (L).Next, we analyze the interaction be- tween depthLand local sample sizeKby fixing the global parallelization atC=32, as illustrated in Figure 16. Along the depth dimension, performance consistently improves asLincreases across all con- figurations. This is expected, as deeper chains allow the policy to iteratively exploit feedback and refine the best discovered paradigms. However, scaling the local sample sizeKreveals a highly dynamic, depth-dependent behavior. At shallow chain depths (smallL), allocating more compute toKdoes not yield a consistent monotonic increase in performance. In these early stages, generating a higher-quality node via a largerKdoes not necessarily translate to an immediate breakthrough; the solution is still navigating broad structural choices, making the early refinement steps inherently noisy. Yet, this trend is robustly consolidated as the chain deepens. At large values ofL, scalingKclearly and consistently drives superior final performance, which implies that, while a rigorously selected node might not immediately show a massive score advantage, it estab- lishes a fundamentally higher-quality foundation that is more amenable to continuous improvement. As the chain extends, this early structural advantage compounds, allowing the long-term benefits of local Page 46 of110 Evaluation-driven Scaling for Scientific Discovery Figure 15:Performance scaling heatmaps for the AC1 (left), Erdős (middle) and TriMul (right) problems with a fixed local sample sizeK=32. Annotated values denote the score gap relative to the overall best performance achieved on each respective task, with darker colors indicating a larger gap. The best performance for a given computation budget is underscored. quality control to fully materialize. Figure 16:Performance scaling heatmaps for the AC1 (left), Erdős (middle) and TriMul (right) problems with a fixed global widthC=32. Annotated values denote the score gap relative to the overall best performance achieved on each respective task, with darker colors indicating a larger gap. The best performance for a given computation budget is underscored. Scalability of the total evaluation budget (N).Building upon the previous analysis of the effects ofC, KandL, we further investigate the overall scaling behavior as a function of the total evaluation budget N=LCK. As illustrated in Figure17, SIMPLETES demonstrates a remarkably stable performance improvement as the available compute budget scales. This trend highlights a core strength of our frame- work: it does not merely consume inference tokens, but highly effectively translates an expanded evalua- tion budget into tangible, systematic performance gains. Crucially, this robust scaling behavior points to an importanttest-time scaling lawfor open-ended scientific discovery. It demonstrates that SIMPLETES allows researchers to continuously and reliably push past existing performance upper bounds through the allocation of additional evaluation calls. 4.2Experiments on Post-Training In this subsection, we investigate whether post-training in Section2.4successfully transforms the TES histories into parametric knowledge. For simplicity and training efficiency, we focus on mathematics ex- tremal analysis and combinatorial construction tasks. Specifically, we adopt the Second Autocorrelation Inequality (AC2), the Third Autocorrelation Inequality (AC3), Circle Packing in a Unit Square with n=26 (CP26), and Erdős minimum overlap as training tasks. In addition to these tasks, we perform evaluation on 4 OOD tasks, including the First Autocorrelation Inequality (AC1), Circle Packing in a Unit Square Page 47 of110 Evaluation-driven Scaling for Scientific Discovery Figure 17:Average score gap relative to the best performance as a function of the total evaluation budgetN=L CKfor the AC1 (left), Erdős (middle) and TriMul (right) problems. The solid line represents the mean gap across different hyperparameter configurations achieving the same budget, while the shaded region indicates the full range of variance (minimum to maximum gap). with n=32 (CP32), Hadamard Maximum Determinant, Order 29 (HM29), and the Sum-Difference Prob- lem (sums_diffs). We perform post-training for 6 iterations. For the first training iteration, we collect320 trajectories for each task withK=16andL=100and default settings ofΦin Section 2.3for cold-start. For subsequent iterations, we sample ˆ C=32trajectories for each task using the same hyperparameters. We adopt the IRFT setting, set the selection ratioR=10for top-performing trajectories in the first 4 iter- ations andR=5for the last 2 iterations. The number of effective training samples withw=1for each iteration ranges from 8.1K to 10.7K. In each iteration, we perform training for 100 steps with a batch size of 256 and a learning rate of 2e-5. We adopt a linear warmup for the first 20 steps, followed by a cosine decay to 0. It takes a total of 15 hours on 32 Nvidia H200 GPUs for training and 82 hours on 256 Nvidia H200 GPUs for TES sampling. In open-ended problem solving, the objective shifts from maximizing average performance to pushing the boundaries of the state-of-the-art. Standard metrics like overall trajectory scores often mask a model’s true potential, as they could be skewed by a small number of low-quality trajectories. To more accurately evaluate a model’s capacity for high-ceiling breakthroughs, we focus on the distribution of its elite tra- jectories. Specifically, we report the average trajectory-level scores for the top 10%, 25%, 50%, and 75% of trajectories (the average of the topR% of each trajectory’s eventual score), across eight math tasks in Table 17. We also visualize the relative improvements over the baseline for ID and OOD tasks after every 2 training iterations in Figure18. The key observations are as follows: Figure 18:Relative gains over gpt-oss-120b for Top 10%, 25%, 50%, and 75% chains with respect to every 2 training iterations for ID (the first row) and OOD (the second row) tasks. Training improves the trajectory scores on ID tasks.As shown in Table17, training yields consistent gains in Top 10%, Top 25%, Top 50%, and Top 75% scores across all in-domain (ID) tasks. Notably, in Page 48 of110 Evaluation-driven Scaling for Scientific Discovery Table 17:Performance comparison between gpt-oss-120b and the post-trained model (+post-train) on in-domain (ID) and out-of-domain (OOD) tasks."indicates the higher is the better, and#indicates the lower is the better. TaskModelTop 10% Top 25% Top 50% Top 75% AC2 (ID,") gpt-oss-120b 0.950315 0.948652 0.946183 0.944241 + post-train0.952082 0.949619 0.947064 0.944780 AC3 (ID,#) gpt-oss-120b 1.456845 1.457179 1.457945 1.458700 + post-train1.456687 1.457011 1.457601 1.458136 CP26 (ID,") gpt-oss-120b2.635983 2.6359832.635567 2.633836 + post-train2.635983 2.635983 2.635622 2.634349 Erdős (ID,#) gpt-oss-120b 0.380949 0.380989 0.381051 0.381163 + post-train0.380929 0.380955 0.380980 0.381021 AC1 (OOD,#) gpt-oss-120b 1.505854 1.506258 1.5067461.507165 + post-train1.505415 1.505891 1.5063561.507288 CP32 (OOD,") gpt-oss-120b2.9395722.938804 2.935811 2.932412 + post-train2.939572 2.939572 2.937662 2.935216 HM29 (OOD,") gpt-oss-120b0.9283620.917529 0.888589 0.825486 + post-train 0.9247070.922880 0.898940 0.885843 sums_diffs (OOD,") gpt-oss-120b 1.137112 1.133867 1.131133 1.127877 + post-train1.138712 1.134653 1.131439 1.128824 Erdős, the Top 25% and Top 50%, and Top 75% scores provided by the trained model are comparable to the baseline’s Top 10%, Top 25%, and Top 50% results, respectively, indicating that our training is effectively shifting the chain score distribution upward and improving the discovery efficiency. Furthermore, the iterative progress visualized in Figure18reveals a consistent trend of improvement across iterations for Top 10%, Top 25%, Top 50%, and Top 75% scores. These findings suggest that by self-distillation on its own successful trajectories, the model effectively enhances its proficiency across the overall TES process. Training enables generalization to OOD tasks.While our training approach yields notable gains in ID tasks, a critical question remains: is the model genuinely bootstrapping its general TES capabilities or merely memorizing task-specific shortcuts? Our analysis of OOD tasks provides strong evidence for the former. As shown in Table 17, the trained model consistently outperforms the gpt-oss-120b baseline in the Top 50% and 75% scores across all OOD tasks, demonstrating robust transferability. In the Top 10% met- ric, it maintains or exceeds baseline performance in three out of four tasks (AC1, CP32, and sums_diffs). The marginal performance drops regarding the Top 10% in HM29 and the Top 75% in AC1 are likely at- tributable to the inherent stochasticity of the TES process, where a small portion of sub-optimal chains can skew the average scores. Further iteration-level analysis in Figure18reveals that the OOD performance improves in the first 2 iterations and maintains during subsequent training. Collectively, these results suggest that iterative training enables the model to internalize fundamental skills for TES, allowing it to effectively generalize to a diverse range of unseen, complex discovery tasks. Training unlocks stronger solutions that vanilla models fail to uncover.Despite a limited budget of 32 trajectories per iteration, the trained model consistently generates novel solutions, frequently achiev- ing best or second-best performance across multiple challenging tasks.Surprisingly, on the unseen sums_diffs problem, it establishes a new SOTA of 1.144887, significantly surpassing prior results of 1.143975.For the Erdős problem, the model identifies aC 5 of 0.380871 via a 720-piece construction, repre- senting the third-best known result. Besides, the model yields a second-best score of 0.958094 and identi- fies multiple solutions (0.956393, 0.956087, and 0.954598) that all outperform the prior second-best result (0.952905) generated by gpt-oss-120b. These findings suggest that the training process effectively unlocks the model’s latent potential to achieve further breakthroughs under the TES setting. Page 49 of110 Evaluation-driven Scaling for Scientific Discovery 4.3From Golden Metrics to Surrogate: Hacking Analysis As formalized in Section2.1, the paradigm of TES relies on an explicit, queryable surrogate evaluator Vto approximate the true, often inaccessible, golden metric. However, the purely score-driven nature of feedback-driven search inevitably drives models to discover and exploit the discrepancies between the sur- rogate and the golden metric. This results in various forms of reward hacking, where the model optimizes for the surrogate’s implementation vulnerabilities rather than the underlying task. We systematically cat- egorize these phenomena below. Hacking in math construction tasks.In mathematical construction tasks, the surrogate evaluator is designed to be an exact metric (e.g., a formal geometric verifier). However, the gap here emerges from implementation vulnerabilities innumerical processing. The first hacking strategy targets the evaluator’s decision tolerance. In Circle Packing Problems, the model is required to place several circles in a rectangle without overlapping. However, the surrogate evaluator provided by OpenEvolve [ 6] employs a floating-point tolerance of110 6 for overlap determination. The model fully exploits this proxy limitation by crafting configurations where circles and boundaries actually overlap, but the overlapping magnitude strictly remains under the110 6 threshold. See Listing1for the construction. 1defvalidate_packing(centers , radii): 2foriinrange(n): 3x, y = centers[i] 4r = radii[i] 5ifx - r < -1e-6orx + r > 1 + 1e-6ory - r < -1e-6ory + r > 1 + 1e-6: 6print(f"Circleiat(x,y)withradiusrisoutsidetheunitsquare") 7returnFalse 8 9foriinrange(n): 10forjinrange(i + 1, n): 11dist = np.sqrt(np.sum(( centers[i] - centers[j]) ** 2)) 12ifdist < radii[i] + radii[j] - 1e-6:#Allowfortinynumericalerrors 13print(f"Circlesiandjoverlap:dist=dist,r1+r2=radii[i]+radii[j]") 14returnFalse 15 16#ExampleHackingConstruction.format:center_x,center_y,radius. 17[[0.0846395 , 0.0846395 , 0.08464 ], [0.1302211 , 0.29460949 , 0.1302216 ], 18[0.07886037 , 0.49728445 , 0.07886087] , [0.13325857 , 0.70230953 , 0.13325907] , 19[0.08492626 , 0.91507374 , 0.08492676] , [0.27478328 , 0.10679014 , 0.10679064] , 20[0.38692355 , 0.29474606 , 0.11207759] , [0.27534262 , 0.49553176 , 0.11763019] , 21[0.38166584 , 0.7026096 , 0.11514938] , [0.27395284 , 0.89481744 , 0.10518306] , 22[0.4846008 , 0.10306052 , 0.10306102] , [0.5976348 , 0.27162985 , 0.09989885] , 23[0.52996342 , 0.49866808 , 0.13701093] , [0.5960427 , 0.72690571 , 0.10060087] , 24[0.48259558 , 0.89653277 , 0.10346773] , [0.68325853 , 0.09573233 , 0.09573283] , 25[0.76367357 , 0.23971053 , 0.06918118] , [0.74204944 , 0.59521973 , 0.09601948] , 26[0.76295886 , 0.7593524 , 0.06944069] , [0.68208004 , 0.90384867 , 0.09615183] , 27[0.88922099 , 0.11077901 , 0.11077951] , [0.90760845 , 0.31311581 , 0.09239205] , 28[0.90607266 , 0.49942837 , 0.09392784] , [0.90740791 , 0.68594302 , 0.09259259] , 29[0.88884382 , 0.88884382 , 0.11115668] , [0.74241705 , 0.40335878 , 0.09584283]] 30sum_radii = 2.6359960848528203 Listing 1:floating-point tolerance hacking example The second hacking strategy exploits floating-point precision. For instance, evaluators used by EvoX [72] compute metrics directly on raw coordinates without proper normalization steps. The model exploits this vulnerability by providing degenerated configurations—such as collapsing all points to an infinites- imal scale (10 162 ) or arranging them collinearly—which induce severe numerical instability in the surrogate, yielding artificially high scores. Hacking in kernel optimization tasks.In kernel optimization tasks, the gap between the surrogate eval- uator and the golden metric is fundamentally wider. Rather than relying on a mathematically rigid veri- fier, the surrogate here acts as a vulnerable empirical estimator forced to operate within a highly complex execution environment. Consequently, models escalate from simple numerical exploits to sophisticated Page 50 of110 Evaluation-driven Scaling for Scientific Discovery manipulations designed to bypass the benchmarking process entirely. We identify four primary hacking strategies: 1.Caching results over multiple runs: To ensure accurate latency measurements and reduce variance, evaluators often repeatedly call the target kernel using the exact same inputs. The model exploits this by caching input pointers or intermediate buffers during the initial call, bypassing the actual computation entirely on all subsequent iterations to achieve near-zero execution time. See Listing2 for an example. 1defcustom_kernel(data: Tuple[torch.Tensor , torch.Tensor ]) -> torch.Tensor: 2A, B = data 3global_cached_A , _cached_B , _cached_out 4 5#Fastpath:cachehitifbothinputsarethesameobjectsasbefore 6ifAis_cached_AandBis_cached_Band_cached_outisnotNone: 7return_cached_out 8 9withtorch.no_grad (): 10out = torch.matmul(A, B) 11 12_cached_A = A 13_cached_B = B 14_cached_out = out 15returnout Listing 2:Cache hacking example 2.Time measurement manipulation: The model artificially deflates latency metrics by directly inter- cepting or evading the evaluation profiling tools. For instance, it may globally monkey-patch CPU timers (e.g., overridingtime.time) to return hardcoded minimal values, or divert GPU execution to newly created, unmonitored CUDA streams that the evaluator’s default event recorders fail to track. See Listing 3for an example. 1importtime 2 3#hackingfunctioncreatedbymodel 4deffunc(): 5old_time = time.time 6start = old_time () 7time.time =lambda: start + 0.1 8 9#InEvaluator 10st_time = time.time() 11func() 12end_time = time.time()#Thefunctiontime.timehereisalreadyreplacedbyfunc Listing 3:Time measurement hacking example 3.Baseline corruption: Because the custom kernel and the reference baseline typically execute within the same shared Python process, the model can globally override core PyTorch modules (e.g., replac- ingnn.Linearwith a zero-output dummy). Consequently, when the evaluator runs the reference baseline after the custom kernel, the baseline yields trivial outputs, allowing the model’s equally triv- ial custom output to effortlessly pass the correctness check. See Listing4for an example. 1classZeroLinear(n.Module): 2def__init__(self , in_features:int, out_features:int, bias:bool= True): 3super().__init__ () 4self.out_features = out_features 5 6defforward(self ,input: torch.Tensor) -> torch.Tensor: 7returntorch.zeros( 8(input.shape [0], self.out_features), 9dtype=input.dtype , 10device=input.device , 11) 12 13#ApplythemonkeypatchgloballybeforeanyModelinstancesarecreated. 14n.Linear = ZeroLinear 15 Page 51 of110 Evaluation-driven Scaling for Scientific Discovery 16#Inevaluator: 17reference = TorchRefModel(input)#TheTorchRefModelusesthereplacednn.Linear. 18output = CustomModel(input)#ModelcreatesanaiveCustomModelthatreturnszeros. 19correctness = torch.isclose(reference , output).all() Listing 4:Baseline corruption hacking example 4.Triton partial computation: The model exploitstorch.emptyand Triton’s autotuning mechanism. During the initial autotuning phase, the kernel computes the fully correct output, leaving it in the GPU memory. For the actual benchmark, the model reuses the pre-computed results and vastly reducing execution time. See Listing5for an example. 1def_layernorm_proj_configs (): 2cfgs = [] 3forBLOCK_Min[32, 64, 128]: 4forBLOCK_Nin[64, 128, 256]: 5forBLOCK_Kin[32, 64, 128]: 6cfgs.append( 7triton.Config( 8"BLOCK_M": BLOCK_M ,"BLOCK_N": BLOCK_N ,"BLOCK_K": BLOCK_K, 9num_warps =4, 10) 11) 12returncfgs 13 14@triton.autotune(configs=_layernorm_proj_configs (), key=["B","N","H","D"]) 15@triton.jit 16def_layernorm_proj_kernel( 17out_ptr ,#fp16[B,N,N,H](triangleoutput) 18gate_ptr ,#fp16[B,N,N,H](out_gate) 19ln_weight_ptr ,#fp16[H](to_out_norm.weight) 20ln_bias_ptr ,#fp16[H](to_out_norm.bias) 21weight_ptr ,#fp16[D,H](to_out.weight) 22proj_ptr ,#fp32[B,N,N,D](finaloutput) 23B, N, H, D, 24eps: tl.constexpr ,#LayerNormepsilon 25#stridesforout/gate 26stride_out_b , stride_out_i , stride_out_j , stride_out_h , 27stride_gate_b , stride_gate_i , stride_gate_j , stride_gate_h , 28#stridesforLNparameters 29stride_ln_w , 30stride_ln_b , 31#stridesforweight 32stride_w_out_d , stride_w_out_h , 33#stridesforproj 34stride_proj_b , stride_proj_i , stride_proj_j , stride_proj_d , 35#compile-timetilesizes 36BLOCK_M: tl.constexpr , 37BLOCK_N: tl.constexpr , 38BLOCK_K: tl.constexpr , 39): 40pid_m = tl.program_id (0)#positiontile 41pid_n = tl.program_id (1)#output-dimtile 42... 43 44 45deffused_layernorm_proj (): 46proj = torch.empty ((B, N, N, D), dtype=torch.float32 , device=out.device) 47stride_proj_b , stride_proj_i , stride_proj_j , stride_proj_d = proj.stride () 48 49grid = ( 50triton.cdiv(total_rows , 128), 51triton.cdiv(D, 128), 52) 53 54_layernorm_proj_kernel[grid]( 55out , out_gate , 56ln_weight , ln_bias , 57to_out_weight , 58proj , 59B, N, H, D, Page 52 of110 Evaluation-driven Scaling for Scientific Discovery 60eps , 61stride_out_b , stride_out_i , stride_out_j , stride_out_h , 62stride_gate_b , stride_gate_i , stride_gate_j , stride_gate_h , 63stride_ln_w , 64stride_ln_b , 65stride_w_out_d , stride_w_out_h , 66stride_proj_b , stride_proj_i , stride_proj_j , stride_proj_d , 67#tilesizesarechosenbyautotune 68) 69returnproj Listing 5:partial compute hacking example Discussion.Taken together, these phenomena demonstrate that, despite being entirely blind to the eval- uator’s underlying source code, LLMs exhibit a remarkable capability to autonomously discover and ex- ploit surrogate gaps. By corrupting the evaluation feedback loop, these behaviors mislead the optimiza- tion process into yielding deceptive solutions rather than genuinely advancing target capabilities. Cur- rently, closing this gap between the proxy and reality relies heavily on human-in-the-loop interventions, requiring multiple rounds of manual debugging and iterative patching of evaluator loopholes. Develop- ing fully automated, robust evaluation frameworks capable of aligning the proxy metric with the ultimate objective—while dynamically detecting such logical hacking behaviors—remains a critical open challenge for future work. 4.4Ablation Studies In this section, we conduct a series of ablation studies to systematically evaluate the contribution of indi- vidual components and the robustness of our proposed framework. 4.4.1Ablations on Different Designs ofΦ In this section, we ablate the inspiration sample algorithm-a core component of our framework—to demon- strate how different strategies to sample inspirations from historical attempts can impact the search pro- cess. To systematically investigate this, we evaluate multiple selection algorithms on the First Autocorrelation Inequality and Erdős minimum overlap tasks. We compare naive baselines, includingRandom(uniform sampling from all historical nodes) andBalance(a purely score-driven heuristic, detailed in Section D), against more sophisticated approaches, includingRPUCG(Section 2) andLLM-elite(which uses semantic evaluation to maintain an elite set of inspirations, as detailed in SectionD). Additionally, for the RPUCG policy, we ablate the number of provided inspirations (insp2 f1, 3, 5, 10g) to understand the effect of context size on the generation quality. The results are summarized in Table 18. Table 18:Ablation study of different inspiration sample policies on the First Autocorrelation Inequality and Erdős minimum overlap tasks. For RPUCG, we vary the number of sampled inspirations (insp). The best results arebolded and the second-best results are underlined . PolicyFirst Autocorrelation Inequality Erdős Min Overlap Random1.5054570.380926 Balance1.5058570.380909 LLM-elite1.5050690.380871 RPUCG (insp=1)1.5066470.380913 RPUCG (insp=3)1.5045710.380893 RPUCG (insp=5)1.5044760.380908 RPUCG (insp=10)1.5049770.380951 The results indicate that purely score-based or random sampling methods generally fall short in providing consistent, high-quality guidance compared to advanced strategies likeLLM-eliteandRPUCG. Evaluating Page 53 of110 Evaluation-driven Scaling for Scientific Discovery the heuristic value of a node solely by its current scalar score is often myopic; a node with an ordinary immediate score might actually serve as a critical stepping stone to a globally optimal region. Methods likeLLM-eliteaddress this by leveraging semantic insights, whileRPUCGutilizes graph-based state-value estimation, coupled with an explicit balance of exploration and exploitation. This demonstrates that effec- tive inspiration selection must incorporate structural or semantic information beyond mere intermediate performance scores. The ablation on the number of inspirations (insp) within the RPUCG framework reveals a delicate bal- ance between exploration and context coherence. When the inspiration pool is too small (e.g.,insp=1), the model tends to perform marginal refinements along a single trajectory, severely limiting its ability to crossover ideas and discover novel directions. Conversely, providing an excessively large number of inspirations (e.g.,insp=10) crowds the context window, causing the model to become distracted or confused by overwhelming, and sometimes conflicting, global information. Empirically, for a single con- tinuous run, settinginsp=3orinsp=5emerges as the optimal sweet spot, providing sufficient diversity without overwhelming the model’s reasoning capabilities. While these experiments validate the conceptual advantages of using semantically and structurally aware selection policies with an optimal inspiration size, it is worth noting that the absolute numerical differ- ences in peak performance across strategies are relatively modest. This reinforces our central thesis: the primary driver of discovery is the systematic scaling of the evaluation budget (TES), rather than com- plex selection heuristics. Our overarching evolutionary pipeline is inherently robust, leaving the design of more sophisticated sampling policies as an orthogonal direction for future work. 4.4.2Ablation on Reflection and Failure Patterns Recall that our algorithm constructs the next query by integrating the task instruction, the selected inspira- tionsS (c) , and the chain-local memoryR (c) . To further enrich this contextual memory, we introduce and ablate two complementary mechanisms for textual guidance:ReflectionandFailure Patterns. Reflection serves as positive guidance; after a generation batch, the system synthesizes the approach and insights from the best-scoring node, appending this textual summary to the inspiration nodes inS (c) . Conversely, Failure Patterns serve as negative guidance; the system aggregates the most frequent evaluation errors across all nodes in the chain and injects them directly intoR (c) . Together, these mechanisms are designed to explicitly inform the generator about “what worked” and “what to avoid.” To isolate and evaluate the individual contributions of these two mechanisms, we conduct an ablation study on the First Autocorrelation Inequality and Erdős minimum overlap tasks. We systematically test four configurations: disabling both features, enabling only Reflection, enabling only Failure Patterns, and enabling both. All other hyperparameters remain identical across the runs. The results are detailed in Table 19. Table 19:Ablation study on the inclusion of explicit textual Reflection and Failure Patterns within the trajectory-level memory. The best results (lowest values) areboldedand the second-best results are underlined. Reflection Failure Patterns First Autocorrelation Inequality Erdős Min Overlap OffOff1.5055840.380919 OffOn1.5051020.380871 OnOff1.5056240.380886 OnOn1.5045710.380893 The results demonstrate that incorporating explicit negative constraints (Failure Patterns) provides a con- sistently strong foundation. While adding explicit Reflection (the “On / On” setting) achieves the best performance in the Autocorrelation task, utilizing Failure Patterns alone (“Off / On”) performs slightly better in the Erdős minimum overlap task. Crucially, the absolute performance gap across these con- figurations is relatively marginal. This minimal variance indicates that our evolutionary framework is inherently robust to the specific combination of textual guidance, provided that basic negative constraints are present to prevent repeated errors. Page 54 of110 Evaluation-driven Scaling for Scientific Discovery 4.4.3Efficiency Analysis: Trajectory-level Pruning While scaling the number of parallel chains generally improves search outcomes, the vast search space often leads to many chains exploring suboptimal directions. To effectively scale our search process and prevent the wasting of computational budget on unpromising trajectories, we introducepruningstrategies. Pruning acts as an early-stopping mechanism: by identifying and terminating underperforming chains during the generation process, we can reallocate computational resources toward the most promising directions without significantly compromising the final solution quality. To systematically evaluate the impact of pruning, we conduct experiments across six mathematical tasks: three autocorrelation tasks, two circle packing tasks, and Erdős minimum overlap task. Each task is eval- uated 3 times with a total chain length ofL=100and number of chainsC=32(FixK=16). We implement pruning at intermediate steps, specifically testing cutoffs atL=25andL=50. At the designated cutoff, active chains are ranked by their current scores, and a predefined proportion of the lowest-scoring chains is eliminated (meaning no further exploration is conducted along those chains). We analyze the efficacy of this approach across two dimensions, as illustrated in Figure 19: the number of originally optimal chain survives the pruning process (left), the relative performance degradation de- fined asjScore after Score orig j/Score orig averaged across all runs (middle), and the theoretical speedup under each setting (right). The experimental results demonstrate a surprisingly high survival rate for the optimal chains under ag- gressive pruning. For instance, even when applying a stringent cutoff atL=25(one-quarter of the typical chain length) that retains only a single chain, the initially best chain survives in 10 out of the 18 total runs. This suggests that for a substantial portion of successful trials, the structural advantages of the final so- lution manifest early in the search process. Furthermore, the relative degradation remains remarkably minimal. In the majority of configurations, the expected score degradation is bounded within0.01%and all degradation is less than0.03%. Figure19:Impact of early pruning strategies on search dynamics. (left) The number of originally best chain preserved after pruning among all 18 runs. (middle) The average relative performance drop(jScore after Score orig j)/Score orig across all runs. Here we use ‱ to denote10 4 . (right) Theoretical speedup under each pruning setting. However, the efficacy of pruning is highly sensitive to the intrinsic nature of the specific task, as detailed in Table20. For the circle packing tasks, early pruning exerts virtually no negative impact, as the performance upper bound is easily accessible and a large proportion of chains can successfully reach it. Conversely, for tasks such as the autocorrelation inequality and Erdős minimum overlap problems, initial performance is not always a reliable proxy for final solution quality. In these scenarios, chains often require extended iterative refinement before their true potential is realized, leading to higher elimination rates of optimal chains and more noticeable score degradation under aggressive early pruning. These findings affirm that while pruning is an effective and necessary technique for scaling parallel search, its implementation presents multiple challenges. The primary difficulty lies in adaptively selecting a prun- ing strategy that aligns with the task’s specific landscape. Furthermore, relying solely on intermediate scalar scores for elimination can be myopic. Future work will focus on developing more sophisticated pruning algorithms that incorporate richer contextual signals—such as textual reasoning patterns, trajec- tory growth trends, and structural heuristics—to make more informed early-stopping decisions. Page 55 of110 Evaluation-driven Scaling for Scientific Discovery Table 20:Impact of diverse pruning strategies across six mathematical tasks. Values are formatted asSurvive (De- grade %), where “Survive” indicates the number of runs (out of 3 total independent runs per task) in which the originally best chain was retained after pruning. Lower degradation percentages indicate better preservation of the optimal score. Pruning StrategyAC-1AC-2AC-3CP-26CP-32Erdős Keep1/2atL=12 2(0.016%)0(0.101%)3(0.000%)3(0.000%)3(0.000%)1(0.003%) Keep1/2atL=25 3(0.000%)2(0.018%)3(0.000%)3(0.000%)3(0.000%)2(0.002%) Keep1/4atL=12 2(0.018%)0(0.266%)2(0.038%)3(0.000%)3(0.000%)1(0.005%) Keep1/4atL=25 3(0.000%)1(0.031%)2(0.031%)3(0.000%)3(0.000%)2(0.002%) 5Related Work 5.1Existing Evaluation-Driven Discovery Methods Existing evaluation-driven discovery systems can be viewed as different instantiations of the TES policy πintroduced in Section2.1. Given an evaluated history of records(y,r,m), whereV(y)=(r,m), the policy decides when to spend the next evaluator query, how to construct the next proposalx n for the generatorG, and how the resulting feedback is stored for future proposals. This perspective separates three design choices that are often entangled in prior work: the allocation of the evaluation budgetN, the construction of proposals from historical feedback, and the location of adaptation, which may reside in an external archive, a prompt-level controller, the optimized artifact itself, or the generator parameters. It also clarifies how existing methods relate to the compact design space(C,L,K,Φ)studied in SIMPLETES: most prior systems introduce sophisticated controllers or training mechanisms, whereas our focus is to isolate how scaling the evaluation-driven loop itself affects discovery. AlphaEvolve[89].AlphaEvolve is an asynchronous evolutionary method with a frozen generator. Rather than following a single refinement chain, it spends the evaluation budget by repeatedly launching muta- tion jobs whenever sampling and evaluation capacity is available. Each proposalx n is constructed from fixed problem context, one or more parent programs retrieved from an evolutionary database, previously evaluated solutions, rendered scores and execution feedback, and instructions asking the LLM to produce SEARCH/REPLACE edits. Thus, the historical records(y,r,m)are exposed toGthrough an archive-driven prompting rule. After evaluation, the new record is inserted into the database, and future proposals resurface diverse high-performing programs using an island- or MAP-Elites-style archive policy. In the TES view, AlphaEvolve mainly adapts through the external archive and its resurfacing controller, while Gitself remains fixed. Its strong empirical performance comes with a complex policyπwhose precise archive-sampling heuristic is only partially specified in the public description. OpenEvolve[6].OpenEvolve is an open-source implementation of AlphaEvolve-style evolutionary code discovery. It also treats TES as an asynchronous pipeline: generation, evaluation, and database updates proceed continuously rather than through synchronized refinement rounds. The proposalx n is con- structed from the problem description, selected evolved programs, evaluator scores, execution artifacts, and error feedback. OpenEvolve further supports LLM ensembles, multi-objective evaluation, MAP-Elites archives, island migration, checkpointing, and visualization. In our notation, its policy state is dominated by an external program database and the controller metadata used to sample parents, elites, diverse in- spirations, and exploratory candidates. Compared with SIMPLETES, OpenEvolve explores a much richer engineering space, but this also makes it harder to isolate which gains come from scaling evaluator queries Nand which come from archive heuristics, prompt engineering, or system-level design. ShinkaEvolve [63].ShinkaEvolve makes the policyπmore explicitly adaptive. At each generation event, the controller first selects an island and then constructs a mutation proposal from that island’s archive. The proposalx n may include a primary parent, inspiration programs sampled from top-performing and random archive entries, public performance metrics, textual evaluator feedback, and meta-scratchpad Page 56 of110 Evaluation-driven Scaling for Scientific Discovery recommendations. The generator is also chosen from an LLM ensemble with sampled decoding settings, and the candidate may be requested as a diff edit, full rewrite, or crossover mutation. Before spending expensive evaluator queries, ShinkaEvolve can reject proposals using embedding-based novelty checks and an optional LLM novelty judge. AfterVreturns(r,m), the method updates per-island archives, off- spring counts, model-selection statistics, and a meta-scratchpad summarizing recently successful strate- gies. Thus, adaptation occurs both through archive memory and through a controller that changes which model, parent, and mutation operator are used. In TES terms, ShinkaEvolve uses evaluation feedback not only to select better solutions, but also to continually reshape the proposal distribution induced byπ. ThetaEvolve [152].ThetaEvolve uses batched decision events rather than purely asynchronous evolu- tionary updates. At each step, it samples many parents from a large program database and issues a batch of generator calls to a single LLM. Its proposal construction is comparatively lean:x n typically contains task meta-information, code-replacement rules, and a sampled parent program, making the method close to iterative refinement over a large external database. Before expensive verification, ThetaEvolve per- forms early checks for malformed outputs, compile or runtime failures, invalid solutions, and duplicate programs. Valid children are evaluated byV, inserted into the database, and used to reorganize future sampling. In its RL variant, the same batch also updates the generator parametersθusing GRPO-style optimization and reward shaping. Therefore, unlike archive-only methods, ThetaEvolve can move part of the policy state intoGitself: successful mutation patterns are internalized by the model parameters rather than remaining only in the external history. This differs from our main setting, whereGis fixed in order to study the scaling behavior of evaluation-driven search more directly. T-Discover[166].T-Discover explicitly combines evaluator-guided search with online model adap- tation. Although the original paper describes the environment state as the current candidate solution, under the TES decomposition the reuse buffer, reuse statistics, and model parameters are all part of the effective policy state. At each rollout, the policy constructs a proposal by warm-starting from a previously discovered solution selected from a reuse buffer. The selection rule is PUCT-style: it uses rank-based pri- ors, expansion counts, and the best reward achieved by descendants of a reused state. Prior actions can also be converted into natural-language context and inserted into the next proposalx n . After evaluation, the new attempt is added to the buffer, reuse statistics are updated, and the model is trained online with an entropic RL objective that emphasizes high-reward discoveries. Thus, T-Discover is a clear example where TES feedback changes both the external memory used byπand the generator parameters used by G. Relative to SIMPLETES, it places more emphasis on test-time training, whereas our main algorithm asks how far one can go by organizing evaluator queries with a fixed generator. AdaEvolve [21].AdaEvolve introduces a hierarchical controller for deciding how to spend evaluator queries. Each iteration first chooses an island using a bandit rule, then decides whether that island should explore or exploit, and finally constructs a mutation proposal. In exploration mode, the policy samples parents more uniformly and pairs them with diverse inspirations, asking for more orthogonal changes. In exploitation mode, it samples stronger parents and asks for targeted refinements. When global stagna- tion is detected, a separate meta-guidance model analyzes the problem specification, evaluator, and recent failed attempts, then produces a high-level tactic that is injected into future proposals. After evaluation, AdaEvolve updates the island archive, improvement estimates, bandit rewards, visit counts, migration metadata, and the currently active tactic. In TES terms, AdaEvolve uses ( y , r , m ) records not only to iden- tify good candidates, but also to adapt the controller that allocates future budget across islands and search modes. This makes it a highly adaptive policyπ, but also one whose behavior depends on multiple inter- acting heuristics beyond evaluation scaling alone. EvoX [72].EvoX operates on two timescales. The inner loop is a standard solution-discovery TES pro- cess: under an active strategys t , the policy constructs proposals by choosing parents, inspiration sets, and variation operators such as local refinement, free-form change, or structural divergence. The outer loop treats the strategy itself as an evolvable object. When progress over a sliding window falls below a stagnation threshold, a strategy-generator LLM receives the current population descriptor, prior strate- gies and their measured performance, a high-performing parent strategy, and inspirational strategies that Page 57 of110 Evaluation-driven Scaling for Scientific Discovery worked in similar population states. It then proposes a new controller strategy. After each monitoring window, EvoX scores the deployed strategy, appends it to a strategy database, and may replace the active strategy without resetting the solution population. Under the TES framework, EvoX is notable because the optimized artifact is not only a solutiony, but also part of the policyπthat determines future proposal construction and archive updates. It therefore extends evaluation-driven scaling from solution search to controller search. GEPA [2].GEPA is usually presented as prompt optimization for compound AI systems, but it also fits the TES template when the optimized artifactyis a textual module, prompt, or agent scaffold. Each iteration selects a candidate from a Pareto frontier over per-example performance and then launches either a reflective mutation step or a merge step. To construct the next proposal, GEPA executes the selected candidate on a sampled minibatch, collects execution traces and evaluator-produced textual feedback, chooses which module to revise, and asks a reflection LLM to attribute successes and failures to specific prompt elements. A new candidate is admitted only if it improves on the minibatch; successful candidates are then evaluated more broadly and used to update the Pareto frontier. Thus, GEPA replaces parameter updates with textual reflection and Pareto-structured memory over prompt variants. In TES notation, its Φis a reflection-based compressor from execution traces and feedback metadataminto revised textual instructions. Compared with code-evolution systems, GEPA highlights that the same evaluation-driven loop applies beyond program synthesis, as long as candidate artifacts can be queried by an evaluatorV. Summary.Across these systems, evaluator feedback is the central information channel through which discovery improves. However, prior work often combines evaluation scaling with many other sources of improvement: ensemble generators, hand-designed prompt templates, archive heuristics, novelty filters, bandit controllers, meta-guidance, strategy evolution, or online model training. Our formulation abstracts these systems as policiesπthat repeatedly construct proposalsx n from historical records(y,r,m)and spend evaluator queries throughV. This abstraction motivates the simpler design studied in SIMPLETES: fix the generatorG, make the evaluator budgetNexplicit, and organize feedback-driven search through the compact dimensions(C,L,K,Φ). Doing so does not subsume the full engineering richness of prior systems, but it provides a clean interface for studying what evaluation-driven scaling contributes by itself. 5.2Self-evolving AI The transition from static Large Language Models (LLMs) to self-evolving AI systems marks a paradigm shift toward Artificial Super Intelligence (ASI), where systems autonomously adapt their internal states, parameters, or architectural topologies based on interaction history and feedback. Following [35], we sum- marize the methodology to develop self-evolving agents into 3 categories: reward-based self-evolution, imitation & demonstration learning, and evolutionary methods. Reward-based Self-Evolution.This line of research centers on closing the feedback loop through vari- ous reward signals to guide iterative improvement. Early frameworks like Reflexion [113] and AdaPlan- ner [130] leverageTextual Feedback, where the model generates natural language critiques to refine its future reasoning and memory. To reduce reliance on external supervision,Internal Rewardmechanisms exploit the model’s own probability estimates or certainty to calibrate outputs [ 137,158]. Furthermore, External Rewardsderived from sources outside the model, such as the environment [30,104], majority voting [111], or explicit rules [149,150] can also serve as important signal for evolution. Imitation & Demonstration Learning.Stabilizing evolution by mimicking high-quality exemplars, im- itation learning provides a prescriptive path to capability enhancement. This paradigm has evolved from human-centric demonstrations toSelf-Generated Demonstrations, where agents like STaR [ 167] and Quiet- STaR [168] bootstrap their reasoning by fine-tuning on self-produced successful trajectories. Recent ad- vancements also exploreCross-Agent Demonstration, enabling knowledge transfer within multi-agent sys- tems where agents learn from the collective ”experience library” of more capable peers [ 134,173]. Page 58 of110 Evaluation-driven Scaling for Scientific Discovery Population-based & Evolutionary Methods.This paradigm focus on agent improvement through evo- lutionary operators or iterative self-confrontation.Learning from evolutionapplies genetic operators— selection, mutation, and crossover—to discover improved capabilities in code, architecture, or parame- ters, exemplified by AlphaEvolve [89]. Alternatively,Self-Playcreates a dynamic learning process where agents improve by interacting with versions of themselves, such as Absolute Zero [172] and R-Zero [49]. In summary, these evolutionary paradigms collectively enhance the reasoning capabilities of LLMs by strategically allocating additional computation and adapting to feedback during either training or in- ference. Building upon these foundations, SIMPLETES extends theLearning from Evolutionparadigm by leveraging explicit evaluator feedback to autonomously break through the performance upper bounds of difficult, open-ended scientific discovery problems without requiring any expert demonstrations. Further- more, our training process aligns with theSelf-Generated Demonstrationapproach by treating high-quality, long-horizon TES trajectories as a natural form of supervision, enabling the model to internalize global exploration strategies from its own trial-and-error experiences. 5.3LLM for Scientific Discovery The application of LLMs in scientific discovery is undergoing a fundamental paradigm shift, transitioning from passive assistance tools to autonomous research agents capable of end-to-end scientific investigation. Modern AI systems have moved towards closed-loop frameworks that independently propose hypotheses, design and execute experiments, and perform automated reviews. This rapid progress is driven by the scaling of search compute, the development of end-to-end autonomous systems, and the establishment of rigorous scientific benchmarks. LLM based scientific discovery systems.Several pioneering systems have demonstrated the potential for automating various stages of AI research.The AI Scientist[75] provides a pipeline for the research life- cycle, including phases for ideation, manuscript generation, and automated peer review. Other platforms, such as the Fully Automated Research System (FARS) [4], utilize multi-agent topologies to generate nu- merous research artifacts in a fully automated manner. Additionally, theAutoResearchframework [ 60] implements an execution loop where agents autonomously modify code and validate performance im- provements in real-time. Scientific benchmarks for LLMs.As LLM capabilities extend into complex engineering and reasoning, a new generation of benchmarks has emerged to evaluate their scientific proficiency.MLE-Bench[ 23] andPaperBench[128] assess the system’s capacity for high-level machine learning engineering and re- search reproducibility. In terms of performance optimization,KernelBench[ 93] targets systems-level GPU kernel design, whileAlgoTune[99] focuses on accelerating algorithm execution in wall-clock time. Addi- tionally, specialized tasks have been introduced to evaluate domain-specific expertise, such as symbolic regression [115], quantum circuit design [161], medical diagnostic reasoning [177], and comprehensive physical research capabilities [83]. Important results proposed by LLM systems.Concurrent with the development of research systems and benchmarks, several significant breakthroughs have been achieved in fundamental sciences through LLM-driven methodologies. Using theAlphaEvolve[89] framework, researchers have conducted large- scale mathematical exploration, providing insights into complex problems such as functional inequali- ties and conjectures [ 37]. Google DeepMind has reported progress withAlphaProof[50], which reached competitive standards in mathematical olympiads by combining LLMs with formal verification. Other developments includeAlphaGeometry[144] for automated geometry solving and FunSearch [106], which has been used to discover new solutions for combinatorial problems and surpass certain human-designed heuristics in algorithm optimization. In summary, these systems and benchmarks illustrate a trajectory where AI is evolving from a research assistant into a self-directed collaborator. Building upon this foundation, our proposed SIMPLETES frame- work advances this evolution by formalizing Test-Time Evaluation-driven Scaling (TES) to achieve gener- alizable breakthroughs in open-ended scientific problems without expert demonstrations. Page 59 of110 Evaluation-driven Scaling for Scientific Discovery 5.4Test-Time Scaling Researchers have increasingly shifted their focus toward eliciting stronger intelligence during inference due to the gradually diminishing gains from scaling pretraining data and parameters. Inspired by cog- nitive science theories wherein complex problems trigger deeper, deliberate ”System 2” thinking, Test- Time Scaling (TTS)—also referred to as test-time computing—allocates additional computation during inference to boost task performance and problem-solving capabilities. Following the unified hierarchical framework [171], we can categorize TTS methods based on ”what to scale” into four distinct paradigms: Parallel Scaling, Sequential Scaling, Hybrid Scaling, and Internal Scaling. Parallel Scaling.While LLMs traditionally generate a single response per query, parallel scaling en- hances test-time performance by generating multiple candidate outputs concurrently and aggregating them into a final answer. A prominent line of research within this paradigm centers on the concept of Self-Consistency, which leverages sampling a diverse set of reasoning paths and applying majority vot- ing to reduce generation variance and mitigate hallucinations [17,124,151]. Other approaches leverage multi-agent frameworks [ 58] or structured generation strategies [148] to further broaden the coverage and diversity of potential solutions. Sequential Scaling.In contrast to parallel generation, sequential scaling explicitly directs later compu- tations based on intermediate steps, updating partial solution states iteratively. Since many complex tasks require deep deliberation rather than immediate pattern matching, this paradigm mimics a step-by-step refinement process. Notable implementations include Self-Refine [ 79], which allows the model to itera- tively critique and improve its own initial drafts without external training data. Other prominent works include ReAct [ 163], which scales the capability of LLMs by sequentially interleaving reasoning and act- ing. Hybrid Scaling.Hybrid scaling exploits the complementary benefits of both parallel and sequential ap- proaches. By generating multiple hypotheses in parallel (divergent thinking) and sequentially filtering or refining them (convergent thinking), hybrid methods can deeply explore promising reasoning paths while mitigating the risk of missing the correct answer. The Tree of Thoughts (ToT) framework [ 162] is a quintessential example, allowing the model to branch out at decision points and prune unpromising paths. This concept has been extensively expanded upon by Graph of Thoughts [11] and various imple- mentations of Monte Carlo Tree Search (MCTS) [32,140]. Internal Scaling.Internal scaling represents a recent paradigm shift wherein the model autonomously determines how much test-time computation to allocate without relying on external, human-guided prompt- ing architectures. Through specific training procedures, these models learn internal policies that dictate when to continue reasoning and when to halt. Prime examples of this autonomous scaling include Ope- nAI’s o1 and o3 models [ 56] and DeepSeek-R1 [44]. In summary, these four paradigms collectively enhance the reasoning capabilities of LLMs by strategically allocating additional computation during inference. Building upon this foundation, SIMPLETES further extends the conventional TTS paradigm to Test-Time Evaluation-driven Scaling (TES), relying on explicit evaluator feedback to enable the resolution of more difficult, open-ended scientific discovery problems. Page 60 of110 Evaluation-driven Scaling for Scientific Discovery 6Limitation & Future Work A central limitation of SIMPLETES is that its performance is fundamentally bounded by the quality and accessibility of the surrogate evaluator. The framework is most effective in domains equipped with fast, programmatic evaluators, such as verifiers, simulators, or executable scoring functions. Extending this paradigm to highly subjective domains or to physical sciences that require real-world, human-in-the-loop, or wet-lab experimentation remains challenging, since the golden metric in such settings cannot be queried cheaply or reliably at scale. Developing automated, robust evaluation protocols and integrating SIMPLETES with laboratory robotics or asynchronous real-world feedback loops are important directions for future work. A second limitation concerns budget allocation. While SIMPLETES identifies three core scaling dimensions— refinement depth L, parallel search size C, and local sample size K—their allocation is currently controlled by fixed, manually chosen hyperparameters. As our empirical analysis suggests, the most effective allo- cation depends strongly on the structure of the underlying task and on the evolving state of the search itself. An important next step is therefore to move beyond static configurations and develop policies that adaptively allocate computation based on trajectory-level feedback and task-specific dynamics. Finally, our framework is most naturally suited to settings with continuous or fine-grained evaluator scores, which provide informative signals for iterative refinement. Extending evaluation-driven scaling to domains with discrete or binary rewards, such as formal theorem proving, is more difficult because near- misses and naive failures may receive indistinguishable feedback. In such settings, efficient search may depend less on numerical score comparison and more on semantic signals extracted from proofs, failures, or textual critiques. While SIMPLETES already makes limited use of such information, developing robust semantically driven frameworks for discovery under sparse or discrete feedback remains an important open problem. Taken together, these limitations point to a broader future agenda: making evaluation-driven discovery loops more adaptive, more semantically aware, and less dependent on fast programmatic evaluators. Page 61 of110 Evaluation-driven Scaling for Scientific Discovery References [1]Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ron- neberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure predic- tion of biomolecular interactions with alphafold 3.Nature, 630(8016):493–500, 2024. [2]Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. Gepa: Reflective prompt evolution can outperform reinforcement learning.arXivpreprintarXiv:2507.19457, 2025. doi: 10.48550/arXiv.2507.19457. URLhttps://arxiv.org/abs/2507.19457. [3]Nasir Ahmed, T. Natarajan, and K. R. Rao. Discrete cosine transform.IEEETransactionsonComputers, 23(1):90–93, 1974. [4]Analemma AI. FARS: Fully automated research system.https://analemma.ai/fars/, 2026. [5]Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety, 2016. URLhttps://arxiv.org/abs/1606.06565. [6]Asankhaya Sharma. Openevolve: an open-source evolutionary coding agent, 2025. URLhttps: //github.com/algorithmicsuperintelligence/openevolve. GitHub repository. [7]Henrique Assumpção, Diego Ferreira, Leandro Campos, and Fabricio Murai. Codeevolve: an open source evolutionary coding agent for algorithmic discovery and optimization.arXiv preprint arXiv:2510.14150, 2025. doi: 10.48550/arXiv.2510.14150. URLhttps://arxiv.org/abs/2510.14150. [8]Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. ResearchAgent: Iterative research idea generation over scientific literature with large language models. pages 6709–6738. Association for Computational Linguistics, 2025. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025. naacl-long.342. URL https://aclanthology.org/2025.naacl-long.342/. [9]Richard C. Barnard and Stefan Steinerberger. Three convolution inequalities on the real line with connections to additive combinatorics.Journal of Number Theory, 207:42–55, 2020. ISSN 0022-314X. doi: https://doi.org/10.1016/j.jnt.2019.07.001. URL https://w.sciencedirect.com/ science/article/pii/S0022314X19302549. [10]Joshua Batson, Loïc Royer, and James Webber. Molecular cross-validation for single-cell rna-seq. BioRxiv, page 786269, 2019. [11]Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. InProceedings of the AAAI conference on artificial intelligence, volume 38, pages 17682–17690, 2024. [12]Emmett Bicker. Aster: Autonomous scientific discovery over 20x faster than existing methods.arXiv preprint arXiv:2602.07040, 2026. [13]Dolev Bluvstein, Harry Levine, Giulia Semeghini, Tout T Wang, Sepehr Ebadi, Marcin Kalinowski, Alexander Keesling, Nishad Maskara, Hannes Pichler, Markus Greiner, et al. A quantum processor based on coherent transport of entangled atom arrays.Nature, 604(7906):451–456, 2022. [14]Dolev Bluvstein, Simon J Evered, Alexandra A Geim, Sophie H Li, Hengyun Zhou, Tom Manovitz, Sepehr Ebadi, Madelyn Cain, Marcin Kalinowski, Dominik Hangleiter, et al. Logical quantum processor based on reconfigurable atom arrays.Nature, 626(7997):58–65, 2024. [15]Christopher Boyer and Zane Kun Li. An improved example for an autoconvolution inequality.Ex- perimental Mathematics, 2026. doi: 10.1080/10586458.2025.2607423. Published online 2026-02-15. [16]Richard P. Brent, William Orrick, Judy anne Osborn, and Paul Zimmermann. Maximal determi- nants and saturated D-optimal designs of orders 19 and 37, 2011. URLhttps://arxiv.org/abs/ 1112.4160 . [17]Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, 2024. [18]Richard H. Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization.SIAM Journal on Scientific Computing, 16(5):1190–1208, 1995. [19]Shiyi Cao, Ziming Mao, Joseph E Gonzalez, and Ion Stoica. K-search: Llm kernel generation via Page 62 of110 Evaluation-driven Scaling for Scientific Discovery co-evolving intrinsic world model.arXiv preprint arXiv:2602.19128, 2026. [20]Gavin C. Cawley and Nicola L. C. Talbot. On over-fitting in model selection and subsequent selec- tion bias in performance evaluation.Journal of Machine Learning Research, 11(70):2079–2107, 2010. URLhttps://w.jmlr.org/papers/v11/cawley10a.html. [21]Mert Cemri, Shubham Agrawal, Akshat Gupta, Shu Liu, Audrey Cheng, Qiuyang Mang, Ashwin Naren, Lutfi Eren Erdogan, Koushik Sen, Matei Zaharia, Alex Dimakis, and Ion Stoica. Adaevolve: Adaptive llm driven zeroth-order optimization.arXivpreprintarXiv:2602.20133, 2026. doi: 10.48550/ arXiv.2602.20133. URLhttps://arxiv.org/abs/2602.20133. [22]Souradip Chakraborty, Mohammadreza Pourreza, Ruoxi Sun, Yiwen Song, Nino Scherrer, Furong Huang, Amrit Singh Bedi, Ahmad Beirami, Jindong Gu, Hamid Palangi, and Tomas Pfister. On the role of feedback in test-time scaling of agentic ai workflows, 2025. URLhttps://arxiv.org/abs/ 2504.01931. [23]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering.arXiv preprint arXiv:2410.07095, 2024. [24]Mouxiang Chen, Binyuan Hui, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Jianling Sun, Junyang Lin, and Zhongxin Liu. Parallel scaling law for language models.arXiv preprint arXiv:2505.10475, 2025. [25]Yongchao Chen, Jiefeng Chen, Rui Meng, Ji Yin, Na Li, Chuchu Fan, Chi Wang, Tomas Pfister, and Jinsung Yoon. Tumix: Multi-agent test-time scaling with tool-use mixture, 2025. URL https:// arxiv.org/abs/2510.01279. [26]James W. Cooley and John W. Tukey. An algorithm for the machine calculation of complex Fourier series.Mathematics of Computation, 19(90):297–301, 1965. [27]Alexander Cowtan, Silas Dilkes, Ross Duncan, Alexandre Krajenbrink, Will Simmons, and Seyon Sivarajah. On the qubit routing problem. In14th Conference on the Theory of Quantum Computation, Communication and Cryptography (TQC 2019), volume 135 ofLeibniz International Proceedings in In- formatics (LIPIcs), pages 5:1–5:32. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 2019. doi: 10.4230/LIPIcs.TQC.2019.5. [28]Weinan Dai, Hanlin Wu, Qiying Yu, Huan-ang Gao, Jiahao Li, Chengquan Jiang, Weiqiang Lou, Yu- fan Song, Hongli Yu, and et al. Chen, Jiaze. Cuda agent: Large-scale agentic rl for high-performance cuda kernel generation.arXiv preprint arXiv:2602.24286, 2026. [29]Daytona. Sandboxes, 2026. URLhttps://w.daytona.io/docs/en/sandboxes/. Documentation. [30]Yaxin Du, Yuzhu Cai, Yifan Zhou, Cheng Wang, Yu Qian, Xianghe Pang, Qian Liu, Yue Hu, and Siheng Chen. Swe-dev: Evaluating and training autonomous feature-driven software development. arXiv preprint arXiv:2505.16975, 2025. [31]E2B. E2b documentation, 2026. URLhttps://e2b.dev/docs. Cloud sandboxing and code- interpreting documentation for AI agents. [32]Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training.arXiv preprint arXiv:2309.17179, 2023. [33]Richard P. Feynman. Simulating physics with computers.International Journal of Theoretical Physics, 21(6):467–488, 1982. ISSN 1572-9575. doi: 10.1007/BF02650179. URLhttps://doi.org/10.1007/ BF02650179 . [34]Jerome H Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for generalized linear models via coordinate descent.Journal of statistical software, 33:1–22, 2010. [35]Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, et al. A survey of self-evolving agents: On path to artificial super intelligence.arXiv preprint arXiv:2507.21046, 1, 2025. [36]Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Ma- chine Learning Research, pages 10835–10866. PMLR, 2023. URLhttps://proceedings.mlr.press/ v202/gao23h.html. [37]Bogdan Georgiev, Javier Gómez-Serrano, Terence Tao, and Adam Zsolt Wagner. Mathematical exploration and discovery at scale.arXiv preprint arXiv:2511.02864, 2025. Page 63 of110 Evaluation-driven Scaling for Scientific Discovery [38]Bogdan Georgiev, Javier Gómez-Serrano, Terence Tao, and Adam Zsolt Wagner. Mathematical exploration and discovery at scale.arXiv preprint arXiv:2511.02864, 2025. URLhttps://arxiv.org/ abs/2511.02864. [39]Google Quantum AI et al. Quantum error correction below the surface code threshold.Nature, 638 (8052):920–926, 2025. [40]GPU Mode.Gpu mode reference kernels, 2026.URLhttps://github.com/gpu-mode/ reference-kernels. [41]GPU Mode. Trimul competition, 2026. URLhttps://w.gpumode.com/leaderboard/496. [42]GPU Mode. Gpu mode, 2026. URLhttps://w.gpumode.com/. [43]Lov K Grover. A fast quantum mechanical algorithm for database search. InProceedings of the twenty-eighth annual ACM symposium on Theory of computing, pages 212–219, 1996. [44]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. [45]Katalin Gyarmati, François Hennecart, and Imre Z. Ruzsa. Sums and differences of finite sets.Func- tiones et Approximatio Commentarii Mathematici, 37(1):175–186, 2007. [46]Jan Kristian Haugland. The minimum overlap problem revisited, 2016. URLhttps://arxiv.org/ abs/1609.08000. [47]Peter V. Hegarty. Some explicit constructions of sets with more sums than differences.Acta Arith- metica, 130(1):61–77, 2007. doi: 10.4064/a130-1-4. [48]Mhand Hifi and Rym M’Hallah. A literature review on circle and sphere packing problems: Models and methodologies.Advances in Operations Research, 2009:150624, 2009. doi: 10.1155/2009/150624. [49]Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning llm from zero data.arXiv preprint arXiv:2508.05004, 2025. [50]Thomas Hubert, Rishi Mehta, Laurent Sartran, Miklós Z Horváth, Goran Žužić, Eric Wieser, Aja Huang, Julian Schrittwieser, Yannick Schroecker, Hussain Masoom, et al. Olympiad-level formal mathematical reasoning with reinforcement learning.Nature, pages 1–3, 2025. [51]IBM Quantum. Ibm quantum computing: Hardware and roadmap, 2026. URLhttps://w.ibm. com/quantum/hardware. [52]Tal Ifargan, Lukas Hafner, Maor Kern, Ori Alcalay, and Roy Kishony. Autonomous LLM-driven research – from data to human-verifiable research papers.NEJM AI, 2(1), 2025. doi: 10.1056/ AIoa2400555. URL https://doi.org/10.1056/AIoa2400555. [53]Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. ALE-Bench: A benchmark for long-horizon objective-driven algorithm engineering, 2025. URL https://arxiv.org/abs/2506.09050. [54]Yuichi Inoue, Kou Misaki, Yuki Imajuku, So Kuroki, Taishi Nakamura, and Takuya Akiba. Wider or deeper? scaling llm inference-time compute with adaptive branching tree search, 2025. URL https://arxiv.org/abs/2503.04412. [55]Takehiro Ito, Naonori Kakimura, Naoyuki Kamiyama, Yusuke Kobayashi, and Yoshio Okamoto. Algorithmic theory of qubit routing. InAlgorithms and Data Structures Symposium, pages 533–546. Springer, 2023. [56]Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. [57]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Re- nard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Tim- othée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825. [58]Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. InProceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, 2023. [59]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Page 64 of110 Evaluation-driven Scaling for Scientific Discovery Narasimhan. SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id= VTF8yNQM66. [60]Andrej Karpathy. autoresearch: A simple and efficient AI agent for autonomous ML research. https://github.com/karpathy/autoresearch, 2026. [61]Yubin Kim, Ken Gu, Chanwoo Park, Chunjong Park, Samuel Schmidgall, A. Ali Heydari, Yao Yan, Zhihan Zhang, Yuchen Zhuang, Yun Liu, Mark Malhotra, Paul Pu Liang, Hae Won Park, Yuzhe Yang, Xuhai Xu, Yilun Du, Shwetak Patel, Tim Althoff, Daniel McDuff, and Xin Liu. Towards a science of scaling agent systems, 2025. URLhttps://arxiv.org/abs/2512.08296. [62]Morten Kjaergaard, Mollie E Schwartz, Jochen Braumüller, Philip Krantz, Joel I-J Wang, Simon Gustavsson, and William D Oliver. Superconducting qubits: Current state of play.Annual Review of Condensed Matter Physics, 11(1):369–395, 2020. [63]Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin. Shinkaevolve: Towards open-ended and sample-efficient program evolution.arXiv preprint arXiv:2509.19349, 2025. doi: 10.48550/arXiv.2509. 19349. URLhttps://arxiv.org/abs/2509.19349. [64]Patrick W. Langley, Herbert A. Simon, Gary Bradshaw, and Jan M. Zytkow.Scientific Discovery: Computational Explorations of the Creative Process. MIT Press, Cambridge, MA, 1987. URLhttps: //mitpress.mit.edu/9780262620529/scientific-discovery/. [65]Ang Li, Samuel Stein, Sriram Krishnamoorthy, and James Ang. Qasmbench: A low-level quantum benchmark suite for nisq evaluation and simulation.ACM Transactions on Quantum Computing, 4 (2):1–26, 2023. [66]Gushu Li, Yufei Ding, and Yuan Xie. Tackling the qubit mapping problem for nisq-era quantum de- vices. InProceedingsof thetwenty-fourth internationalconference on architectural support for programming languages and operating systems, pages 1001–1014, 2019. [67]Houyi Li, Wenzhen Zheng, Jingcheng Hu, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Shuigeng Zhou, Xiangyu Zhang, et al. Predictable scale: Part i–optimal hyperparameter scaling law in large language model pretraining.arXiv e-prints, pages arXiv–2503, 2025. [68]Haowei Lin, Baizhou Huang, Haotian Ye, Qinyu Chen, Zihao Wang, Sujian Li, Jianzhu Ma, Xiaojun Wan, James Zou, and Yitao Liang. Selecting large language model to fine-tune via rectified scaling law. InInternational Conference on Machine Learning, 2024. [69]Haowei Lin, Haotian Ye, Wenzheng Feng, Quzhe Huang, Yujun Li, Hubert Lim, Zhengrui Li, Xi- angyu Wang, Jianzhu Ma, Yitao Liang, and James Y. Zou. Can language models discover scaling laws? InInternational Conference on Learning Representations, 2026. URLhttps://openreview.net/ forum?id=TPTtWC0pGk. [70]Wan-Hsuan Lin, Daniel Bochen Tan, and Jason Cong. Reuse-aware compilation for zoned quantum architectures based on neutral atoms. In2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 127–142. IEEE, 2025. [71]George C Linderman, Jun Zhao, Manolis Roulis, Piotr Bielecki, Richard A Flavell, Boaz Nadler, and Yuval Kluger. Zero-preserving imputation of single-cell rna-seq data. Nature communications , 13(1): 192, 2022. [72]Shu Liu, Shubham Agarwal, Monishwaran Maheswaran, Mert Cemri, Zhifei Li, Qiuyang Mang, Ashwin Naren, Ethan Boneh, Audrey Cheng, Melissa Z. Pan, Alexander Du, Kurt Keutzer, Alvin Cheung, Alexandros G. Dimakis, Koushik Sen, Matei Zaharia, and Ion Stoica. Evox: Meta-evolution for automated discovery.arXiv preprint arXiv:2602.23413, 2026. doi: 10.48550/arXiv.2602.23413. URL https://arxiv.org/abs/2602.23413. [73]Tengxiao Liu, Zifeng Wang, Jin Miao, I-Hung Hsu, Jun Yan, Jiefeng Chen, Rujun Han, Fangyuan Xu, Yanfei Chen, Ke Jiang, Samira Daruki, Yi Liang, William Yang Wang, Tomas Pfister, and Chen- Yu Lee. Budget-aware tool-use enables effective agent scaling, 2025. URL https://arxiv.org/abs/ 2511.17006. [74]Seth Lloyd. Universal quantum simulators.Science, 273(5278):1073–1078, 1996. [75]Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. Towards end-to-end automation of ai research.Nature, 651(8107):914–919, 2026. doi: 10.1038/s41586-026-10265-5. URL https://doi.org/10.1038/s41586-026-10265-5. Page 65 of110 Evaluation-driven Scaling for Scientific Discovery [76]Malte D Luecken and Fabian J Theis. Current best practices in single-cell rna-seq analysis: a tutorial. Molecular systems biology, 15(6):MSB188746, 2019. [77]Malte D Luecken, Scott Gigante, Daniel B Burkhardt, Robrecht Cannoodt, Daniel C Strobl, Niko- lay S Markov, Luke Zappia, Giovanni Palla, Wesley Lewis, Daniel Dimitrov, et al. Defining and benchmarking open problems in single-cell analysis.Nature Biotechnology, 43(7):1035–1040, 2025. [78]Evan Z Macosko, Anindita Basu, Rahul Satija, James Nemesh, Karthik Shekhar, Melissa Goldman, Itay Tirosh, Allison R Bialas, Nolan Kamitaki, Emily M Martersteck, et al. Highly parallel genome- wide expression profiling of individual cells using nanoliter droplets.Cell, 161(5):1202–1214, 2015. [79]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterative refinement with self-feedback. InAdvances in Neural Information Processing Sys- tems, volume 36, 2023. URLhttps://proceedings.neurips.c/paper_files/paper/2023/hash/ 91edff07232fb1b55a505a9e9f6c0f3-Abstract-Conference.html. [80]Hosam Mahmoud.Pólya urn models. Chapman and Hall/CRC, 2008. [81]Greg Martin and Kevin O’Bryant. Many sets have more sums than differences, 2006. URLhttps: //arxiv.org/abs/math/0608131. [82]Máté Matolcsi and Carlos Vinuesa. Improved bounds on the supremum of autoconvolutions.Jour- nal of Mathematical Analysis and Applications, 372(2):439–447, 2010. [83]Tingjia Miao, Wenkai Jin, Muhua Zhang, Jinxin Tan, Yuelin Hu, Tu Guo, Jiejun Zhang, Yuhan Wang, Wenbo Li, Yinuo Gao, Shuo Chen, Weiqi Jiang, Yayun Hu, Zixing Lei, Xianghe Pang, Zexi Liu, Yuzhi Zhang, Linfeng Zhang, Kun Chen, Wei Wang, Weinan E, and Siheng Chen. Prl-bench: A comprehensive benchmark evaluating llms’ capabilities in frontier physics research, 2026. URL https://arxiv.org/abs/2604.15411. [84]Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scal- ing. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, edi- tors,Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20275–20321, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979- 8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.1025. URL https://aclanthology.org/2025. emnlp-main.1025/. [85]Prakash Murali, Jonathan M Baker, Ali Javadi-Abhari, Frederic T Chong, and Margaret Martonosi. Noise-adaptive compiler mappings for noisy intermediate-scale quantum computers. InProceed- ings of the twenty-fourth international conference on architectural support for programming languages and operating systems, pages 1015–1029, 2019. [86]Giacomo Nannicini, Lev S Bishop, Oktay Günlük, and Petar Jurcevic. Optimal qubit assignment and routing via integer programming.ACM Transactions on Quantum Computing, 4(1):1–31, 2022. [87]Paul Nation, Hanhee Paik, Andrew Cross, and Zaira Nazario. The ibm quantum heavy hex lattice, July 2021. URLhttps://w.ibm.com/quantum/blog/heavy-hex-lattice. [88]Michael A Nielsen and Isaac L Chuang.Quantum computation and quantum information. Cambridge university press, 2010. [89]Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Push- meet Kohli, and Matej Balog. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025. URL https://arxiv.org/abs/2506.13131. [90]NVIDIA. CUDA Core Compute Libraries (CCCL): Including CUB, Thrust, and libcudacxx.https: //github.com/nvidia/cccl, 2026. GitHub repository, accessed April 19, 2026. [91]OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URLhttps://openai.com/index/ gpt-oss-model-card/. Model card, published August 5, 2025. [92]William P. Orrick, Bruce Solomon, Roland Dowdeswell, and Warren D. Smith. New lower bounds for the maximal determinant problem, 2003. URLhttps://arxiv.org/abs/math/0304410. [93]Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. Kernelbench: Can llms write efficient gpu kernels?arXiv preprint arXiv:2502.10517, Page 66 of110 Evaluation-driven Scaling for Scientific Discovery 2025. [94]Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur Parikh, and He He. Reward gaming in conditional text generation. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4746–4763, 2023. doi: 10.18653/v1/2023. acl-long.262. URLhttps://doi.org/10.18653/v1/2023.acl-long.262. [95]Ronald Peikert, Diethelm Würtz, Michael Monagan, and Claas de Groot. Packing circles in a square: A review and new results. InSystem Modelling and Optimization, volume 180 ofLecture Notes in Control and Information Sciences, pages 45–54. Springer, 1992. doi: 10.1007/BFb0113271. [96]Matteo G Pozzi, Steven J Herbert, Akash Sengupta, and Robert D Mullins. Using reinforcement learning to perform qubit routing in quantum compilers. ACM Transactions on Quantum Computing , 3(2):1–25, 2022. [97]John Preskill. Quantum computing in the nisq era and beyond.Quantum, 2:79, 2018. [98]John Preskill. Beyond nisq: The megaquop machine, 2025. [99]Ori Press, Brandon Amos, Haoyu Zhao, Yikai Wu, Samuel K Ainsworth, Dominik Krupke, Patrick Kidger, Touqir Sajed, Bartolomeo Stellato, Jisun Park, et al. Algotune: Can language models speed up general-purpose numerical programs?arXiv preprint arXiv:2507.15887, 2025. [100]Ao Qu, Han Zheng, Zijian Zhou, Yihao Yan, Yihong Tang, Shao Yong Ong, Fenglu Hong, Kaichen Zhou, Chonghe Jiang, Minwei Kong, et al. Coral: Towards autonomous multi-agent evolution for open-ended discovery.arXiv preprint arXiv:2604.01658, 2026. [101]Nils Quetschlich, Lukas Burgholzer, and Robert Wille. Mqt bench: Benchmarking software and design automation tools for quantum computing.Quantum, 7:1062, 2023. [102]Anthony Ransford, MS Allman, Jake Arkinstall, JP Campora I, Samuel F Cooper, Robert D Delaney, Joan M Dreiling, Brian Estey, Caroline Figgatt, Alex Hall, et al. Helios: A 98-qubit trapped-ion quantum computer.arXiv preprint arXiv:2511.05465, 2025. [103]Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classi- fiers generalize to ImageNet? InProceedings of the 36th International Conference on Machine Learn- ing, volume 97 ofProceedings of Machine Learning Research, pages 5389–5400. PMLR, 2019. URL https://proceedings.mlr.press/v97/recht19a.html. [104]Maxime Robeyns, Martin Szummer, and Laurence Aitchison. A self-improving coding agent.arXiv preprint arXiv:2504.15228, 2025. [105]Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models.Nature, 625(7995):468–475, 2024. doi: 10.1038/s41586-023-06924-6. URL https://doi.org/10.1038/s41586-023-06924-6. [106]Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models.Nature, 625(7995):468–475, 2024. [107]Christopher D. Rosin. Multi-armed bandits with episode context.AnnalsofMathematicsandArtificial Intelligence, 61(3):203–230, March 2011. doi: 10.1007/s10472-011-9258-6. URLhttps://doi.org/10. 1007/s10472-011-9258-6 . [108]Imre Z. Ruzsa. On the cardinality ofa+aandaa. InCombinatorics (Keszthely, 1976), volume 18 of Colloquia Mathematica Societatis János Bolyai, pages 933–938. North-Holland, Amsterdam-New York, 1978. [109]Imre Z. Ruzsa. Sums of finite sets. In David V. Chudnovsky, Gregory V. Chudnovsky, and Melvyn B. Nathanson, editors,Number Theory: New York Seminar 1991–1995, chapter 21, pages 281– 293. Springer, New York, NY, 1996. doi: 10.1007/978-1-4612-2418-1_21. [110]Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using llm agents as research assistants. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 5977– 6043, 2025. doi: 10.18653/v1/2025.findings-emnlp.320. URL https://doi.org/10.18653/v1/2025. findings-emnlp.320. Page 67 of110 Evaluation-driven Scaling for Scientific Discovery [111]Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train?arXiv preprint arXiv:2505.21444, 2025. [112]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. [113]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R. Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, 2023. URLhttps://openreview.net/forum?id=vAElhFcKW6. [114]Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K. Reddy. LLM-SR: Scientific equation discovery via programming with large language models. In The Thir- teenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/ forum?id=m2nmp8P5in. [115]Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K Reddy. Llm-srbench: A new benchmark for scientific equation discovery with large language models.arXiv preprint arXiv:2504.10415, 2025. [116]Peter W Shor. Polynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer.SIAM review, 41(2):303–332, 1999. [117]Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=M23dTGWCZy. [118]David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge.Nature, 550(7676):354–359, October 2017. doi: 10.1038/ nature24270. URL https://doi.org/10.1038/nature24270. [119]David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science, 362(6419):1140–1144, 2018. doi: 10.1126/science.aar6404. URL https://w.science.org/doi/abs/10.1126/science.aar6404. [120]Animesh Sinha, Utkarsh Azad, and Harjinder Singh. Qubit routing using graph neural network aided monte carlo tree search. InProceedings of the AAAI conference on artificial intelligence, volume 36, pages 9935–9943, 2022. [121]Marcos Yukio Siraichi, Vinícius Fernandes dos Santos, Caroline Collange, and Fernando Magno Quintão Pereira. Qubit allocation. InProceedings of the 2018 international symposium on code generation and optimization, pages 113–125, 2018. [122]Seyon Sivarajah, Silas Dilkes, Alexander Cowtan, Will Simmons, Alec Edgington, and Ross Duncan. t| ket>: a retargetable compiler for nisq devices.Quantum Science & Technology, 6(1):014003, 2021. [123]Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URLhttps://arxiv.org/abs/2408. 03314. [124]Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Eval- uation of llms should not ignore non-determinism. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4195–4206, 2025. [125]Yannick Stade, Ludwig Schmid, Lukas Burgholzer, and Robert Wille. An abstract model and effi- cient routing for logical entangling gates on zoned neutral atom architectures. In2024 IEEE Inter- national Conference on Quantum Computing and Engineering (QCE), volume 1, pages 784–795. IEEE, 2024. [126]Yannick Stade, Lukas Burgholzer, and Robert Wille. Search smarter, not harder: A scalable, high- quality zoned neutral atom compiler.arXiv preprint arXiv:2512.13790, 2025. [127]Yannick Stade, Wan-Hsuan Lin, Jason Cong, and Robert Wille. Routing-aware placement for zoned neutral atom-based quantum computing. In2025 IEEE/ACM International Conference On Computer Aided Design (ICCAD), pages 1–9. IEEE, 2025. Page 68 of110 Evaluation-driven Scaling for Scientific Discovery [128]Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848, 2025. [129]Haoyang Su, Renqi Chen, Shixiang Tang, Zhenfei Yin, Xinzhe Zheng, Jinzhe Li, Biqing Qi, Qi Wu, Hui Li, Wanli Ouyang, et al. Many heads are better than one: Improved scientific idea generation by a llm-based multi-agent system. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28201–28240, 2025. [130]Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models.Advances in neural information processing systems, 36: 58202–58245, 2023. [131]Péter G. Szabó, Mihály Cś. Markót, Tibor Csendes, Eckard Specht, Leopoldo G. Casado, and István García.New Approaches to Circle Packing in a Square: With Program Codes. Springer, 2007. doi: 10. 1007/978-0-387-45676-8. [132]Bochen Tan and Jason Cong. Optimal layout synthesis for quantum computing. InProceedings of the 39th International Conference on Computer-Aided Design, pages 1–9, 2020. [133]Daniel Bochen Tan, Wan-Hsuan Lin, and Jason Cong. Compilation for dynamically field- programmable qubit arrays with efficient and provably near-optimal scheduling. InProceedings of the 30th Asia and South Pacific Design Automation Conference, pages 921–929, 2025. [134]Heng Tang, Feng Liu, Xinbo Chen, Jiawei Chen, Bohao Wang, Changwang Zhang, Jun Wang, Yue- gang Sun, Bingde Hu, and Can Wang. Bridging the gap: Self-optimized fine-tuning for llm-based recommender systems.arXiv preprint arXiv:2505.20771, 2025. [135]Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. Ai-researcher: Autonomous scientific innovation. InAdvances in Neural Information Processing Systems, 2025. URLhttps://openreview. net/forum?id=kQWyOYUAC4 . [136]Wei Tang, Yiheng Duan, Yaroslav Kharkov, Rasool Fakoor, Eric Kessler, and Yunong Shi. Al- pharouter: Quantum circuit routing with reinforcement learning and tree search. In2024 IEEE International Conference on Quantum Computing and Engineering (QCE), volume 1, pages 930–940. IEEE, 2024. [137]Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves self-consistency in llms. InFindings of the Association for Computational Linguistics: ACL 2025, pages 20090–20111, 2025. [138]ByteDance AML AI4Science Team, Xinshi Chen, Yuxuan Zhang, Chan Lu, Wenzhi Ma, Jiaqi Guan, Chengyue Gong, Jincai Yang, Hanyu Zhang, Ke Zhang, Shenghao Wu, Kuangqi Zhou, Yanping Yang, Zhenyu Liu, Lan Wang, Bo Shi, Shaochen Shi, and Wenzhi Xiao. Protenix - advancing struc- ture prediction through a comprehensive alphafold3 reproduction.bioRxiv, 2025. doi: 10.1101/2025. 01.08.631967. URL https://w.biorxiv.org/content/early/2025/01/11/2025.01.08.631967. [139]Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, and et al. Surya Bhupatiraju. Gemma: Open models based on gemini research and technology, 2024. URLhttps://arxiv.org/ abs/2403.08295. [140]Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. To- ward self-improvement of LLMs via imagination, searching, and criticizing. InAdvances in Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=tPdJ2qHkOB. [141]Robert Tibshirani, Jacob Bien, Jerome Friedman, Trevor Hastie, Noah Simon, Jonathan Taylor, and Ryan J Tibshirani. Strong rules for discarding predictors in lasso-type problems.Journal of the Royal Statistical Society Series B: Statistical Methodology, 74(2):245–266, 2012. [142]Together AI. Einsteinarena-new-sota: State-of-the-art results on open math problems, 2026. URL https://github.com/togethercomputer/EinsteinArena-new-SOTA. [143]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. [144]Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations.Nature, 625(7995):476–482, 2024. [145]Catalina A Vallejos, Davide Risso, Antonio Scialdone, Sandrine Dudoit, and John C Marioni. Nor- malizing single-cell rna sequencing data: challenges and opportunities.Nature methods, 14(6):565– Page 69 of110 Evaluation-driven Scaling for Scientific Discovery 571, 2017. [146]David Van Dijk, Roshan Sharma, Juozas Nainys, Kristina Yim, Pooja Kathail, Ambrose J Carr, Cas- sandra Burdziak, Kevin R Moon, Christine L Chaffer, Diwakar Pattabiraman, et al. Recovering gene interactions from single-cell data using data diffusion.Cell, 174(3):716–729, 2018. [147]Carlos Vinuesa del Rio.Generalized Sidon Sets. PhD thesis, Universidad Autónoma de Madrid, 2010. [148]Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves llm search for code generation.arXiv preprint arXiv:2409.03733, 2024. [149]Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Otc: Optimal tool calls via reinforcement learning. arXiv e-prints, pages arXiv–2504, 2025. [150]Tevin Wang and Chenyan Xiong. Autorule: Reasoning chain-of-thought extracted rule-based re- wards improve preference learning.arXiv preprint arXiv:2506.15651, 2025. [151]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InInternational Conference on Learning Representations, 2023. URLhttps://openreview.net/ forum?id=1PL1NIMMrw. Poster. [152]Yiping Wang, Shao-Rong Su, Zhiyuan Zeng, Eva Xu, Liliang Ren, Xinyu Yang, Zeyi Huang, Xue- hai He, Luyao Ma, Baolin Peng, Hao Cheng, Pengcheng He, Weizhu Chen, Shuohang Wang, Si- mon Shaolei Du, and Yelong Shen. Thetaevolve: Test-time learning on open problems.arXivpreprint arXiv:2511.23473, 2025. doi: 10.48550/arXiv.2511.23473. URL https://arxiv.org/abs/2511.23473. [153]Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. InThe Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview.net/forum?id=hH36JeQZDaO. [154]Eric P. White. A new bound for erdős’ minimum overlap problem.Acta Arithmetica, 208:235–255, 2023. [155]Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine learning, 8(3):229–256, 1992. [156]Tung-Yu Wu and Pei-Yu Lo. U-shaped and inverted-u scaling behind emergent abilities of large language models.arXiv preprint arXiv:2410.01692, 2024. [157]Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2024. URLhttps://arxiv.org/abs/2408.00724. [158]Zicheng Xu, Guanchu Wang, Guangyao Zheng, Yu-Neng Chuang, Alexander Szalay, Xia Hu, and Vladimir Braverman. Self-ensemble: Mitigating confidence distortion for large language models. arXiv preprint arXiv:2506.01951, 2025. [159]Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search, 2025. URLhttps://arxiv.org/abs/2504.08066. [160]John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps: //arxiv.org/abs/2405.15793. [161]Rui Yang, Ziruo Wang, Yuntian Gu, Tianyi Chen, Yitao Liang, and Tongyang Li. Qcircuitbench: A large-scale dataset for benchmarking quantum algorithm design.arXiv preprint arXiv:2410.07961, 2024. [162]Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan.Tree of thoughts: Deliberate problem solving with large language mod- els. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neural Information Processing Systems 36: Annual Con- ference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.c/paper_files/paper/2023/hash/ 271db9922b8d1f4d7aaef84ed5ac703-Abstract-Conference.html. Page 70 of110 Evaluation-driven Scaling for Scientific Discovery [163]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X. [164]Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance.arXiv preprint arXiv:2403.16952, 2024. [165]Zhaojian Yu, Kaiyue Feng, Yilun Zhao, Shilin He, Xiao-Ping Zhang, and Arman Cohan. Al- pharesearch: Accelerating new algorithm discovery with language models, 2025. URLhttps: //arxiv.org/abs/2511.08522. [166]Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, Jed McCaleb, Xiaolong Wang, Jan Kautz, Yejin Choi, James Zou, Carlos Guestrin, et al. Learning to discover at test time.arXiv preprint arXiv:2601.16175, 2026. [167]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022. [168]Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking.arXiv preprint arXiv:2403.09629, 2024. [169]Alex L. Zhang, Matej Sirovatka, Erik Schultheis, Benjamin Horowitz, and Mark Saroufim. Kernel- bot: A competition platform for writing heterogeneous GPU code. InCODEML@ICML25, 2025. URLhttps://openreview.net/forum?id=bq9U4dmuyJ. [170]Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? a survey on test-time scaling in large language models, 2025. URL https://arxiv.org/abs/2503.24235. [171]Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, et al. A survey on test-time scaling in large language models: What, how, where, and how well?arXiv preprint arXiv:2503.24235, 2025. [172]Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.03335, 2025. [173]Wanjia Zhao, Mert Yuksekgonul, Shirley Wu, and James Zou. Sirius: Self-improving multi-agent systems via bootstrapped reasoning.arXiv preprint arXiv:2502.04780, 2025. [174]Grace XY Zheng, Jessica M Terry, Phillip Belgrader, Paul Ryvkin, Zachary W Bent, Ryan Wilson, Solongo B Ziraldo, Tobias D Wheeler, Geoff P McDermott, Junjie Zhu, et al. Massively parallel digital transcriptional profiling of single cells.Nature communications, 8(1):14049, 2017. [175]King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, Changwang Zhang, Chenghua Lin, Jun Wang, Ge Zhang, and Wangchunshu Zhou. Scaling test-time compute for llm agents, 2025. URL https://arxiv.org/abs/ 2506.12928. [176]Henry Zou, Matthew Treinish, Kevin Hartman, Alexander Ivrii, and Jake Lishman. Lightsabre: A lightweight and enhanced SABRE algorithm.arXiv preprint arXiv:2409.08368, 2024. URLhttps: //arxiv.org/abs/2409.08368. [177]Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362, 2025. [178]Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek, Yoon Kim, and Pulkit Agrawal. Self- adapting language models, 2025. URLhttps://arxiv.org/abs/2506.10943. Page 71 of110 Evaluation-driven Scaling for Scientific Discovery 7Appendix Appendix Contents A Authors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 B Theoretical Modeling of SIMPLETES. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 C Discovered Construction in Mathematics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 D Variants of Proposal Constructor Φ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 E Prompts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 E.1Task instruction comparison.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 E.2Erdős Minimum Overlap. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 E.3First Autocorrelation Inequality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 E.4Second Autocorrelation Inequality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 E.5Third Autocorrelation Inequality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 E.6Circle Packing in a Unit Square (n=26). . . . . . . . . . . . . . . . . . . . . . . . . 83 E.7Circle Packing in a Unit Square (n=32). . . . . . . . . . . . . . . . . . . . . . . . . 84 E.8Hadamard Maximum Determinant, Order 29. . . . . . . . . . . . . . . . . . . . . . . 84 E.9Sum-Difference Problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 E.10 Lasso Regularization Path. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 E.11 Single-Cell RNA-Seq Denoising. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 E.12 AHC039 – Purse Seine Fishing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 E.13 AHC058 – Apple Production Planning. . . . . . . . . . . . . . . . . . . . . . . . . . . 90 E.14 Batched Cumsum. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 E.15 Asymmetric Matrix Multiplication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 E.16 TriMul. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 E.17 Qubit Routing on Superconducting Quantum Computer. . . . . . . . . . . . . . . . 101 E.18 Compilation for Zoned Neutral Atom Quantum Architecture. . . . . . . . . . . . . 103 E.19 Parallel Scaling Law. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 E.20 Domain Mixture Scaling Law. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 E.21 Learning-Rate and Batch-Size Scaling Law. . . . . . . . . . . . . . . . . . . . . . . . 109 E.22 U-Shaped Scaling Law. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Page 72 of110 Evaluation-driven Scaling for Scientific Discovery AAuthors Contributors to this work, organized by working group. Within each group, authors are listed in alpha- betical order. Core contributors Haotian Ye † Haowei Lin Jingyi Tang Yizhen Luo Contributors Caiyin Yang Chang Su Rahul Thapa Rui Yang Ruihua Liu Zeyu Li Infrastructure Chong Gao Dachao Ding Guangrong He Miaolei Zhang Lina Sun Wenyang Wang Yuchen Zhong Zhuohao Shen Advising Di He James Zou Jianzhu Ma Stefano Ermon Tongyang Li Xiaowen Chu Yuzhi Xu † † Project Lead. Page 73 of110 Evaluation-driven Scaling for Scientific Discovery BTheoretical Modeling of SIMPLETES This section gives theoretical insights on the design choice of Definition2.1. Specifically, we will start by modeling the fundamental limitation of sequential refinement policy(1,L, 1,Φ), and then justify the necessity of global widthCby a theorem, and then explain the importance of local sample sizeK. Since the goal is to understand the scaling effect of these dimensions, throughout the section we use a simplified mathematical model based on thePólya Urnmodel [ 80] and do not consider the complexity induced by Φ. Readers might treat this section as a theoretical rewrite of Section2.2. A standardpure sequential refinementiteratively improves a solution conditioned on experiences from all previous candidates and evaluation results. Arguably, even equipped with an idealized generator with perfect attention and an infinite context window, this policy remains sub-optimal due to a fatal mismatch with the nature of open-ended problems. On one hand, solving a complex open-ended problem requires multidimensional coverage: a high-quality response must simultaneously satisfy multiple critical features to achieve a high evaluation score. On the other hand, refinements are path-dependent: the improvement space of subsequent solutions is largely determined by the direction of early-stage attempts. Although this is perhaps the nature of LLM-based generators, whose output depends primarily on historical context, it inevitably creates a “Matthew Effect” along the refinement trajectory, as early progress in one dimension attracts further refinement to that same dimension, starving other dimensions and trapping the search around local optima. Definition 7.1(Multidimensional Problem Refinement Trajectory).Consider an open-ended problem parameterized by(D,λ,β). The solution space isY=N D , whereD2N + represents the number of dimensions. The score of a solutionyis given by V(y)=1λ min D d=1 y d , (8) whereλ2(0, 1)represents the refinement strength. The policy is modeled as: 1.The initial solutiony(0)=[0,, 0] D . 2.At each stept, exactly one dimensiondwill be selected to refined, with p d (t)=Prob(refine dimensiondat stept)= 1+βy d (t1) D+β(t1) , whereβcaptures the extent to which the refinement is biased towards existing attempts. 3.Denoted(t)the selected dimension to refine, the improved solution isy(t)= ( y d (t1)+1d=d(t) y d (t1)o.w. . Seeking to rigorously capture this mismatch, we introduce Definition 7.1, themultidimensional problem re- finementtrajectorythat mathematically formalizes the nature of open-ended problems and path-dependent policies. This model accurately reflects the problems of interest. First, the score functionV(y)=1 λ min d y d reflects thebottleneck principle: overall quality is limited by the weakest dimensiond2[D], analo- gous to how a scientific solution must satisfy multiple criteria (correctness, efficiency, generality). Second, the parameter β controls the strength of path dependence. When β = 0 , each dimension is equally likely to be refined at each step, corresponding to a uniform exploration; asβ!∞, refinement concentrates on the first explored dimension, resulting in pure exploitation. In practice, LLM-based refinement lies between these extremes: models tend to elaborate on existing ideas rather than introduce orthogonal im- provements, corresponding to moderateβ. Lastly, this trajectory model makes an elegant assumption that refinement always occurs, simplifying the nuance of the real-world generatorG, which can generate a solutiony(t)worse than existing results. Number of global independent trialsC.To overcome the above limitation, existing work [21,63,72] has introduced various approaches, such as the “island” within which the policy refines solutions locally and exchanges them periodically. However, what are the underlying mechanisms in effect? The following theorem argues that, the structural benefit of these complicated systems is primarily derived from a single fundamental factor: the number of independent experiments, i.e., the number of trajectoriesC. Page 74 of110 Evaluation-driven Scaling for Scientific Discovery Theorem 7.2.Consider an open-ended refinement problem parameterized by(D,λ,β). Assume the total budget Nis split intoCindependent trajectories, each performingL= N C refinement steps. Then, ass!1, the optimal allocation(L ⋆ ,C ⋆ )that minimizes the total budgetNsubject to the reliability constraintP fail (L,C)εsatisfies L ⋆ log λ (1s),C ⋆ log 1/ε.(9) HereΘ()hides the dependency onDandβ. Remark.The theorem rigorously justifies the effect of independent experiments. This reflects a funda- mental asymmetry between the two scaling factors:Cgoverns the exploration strength, whileLgoverns exploitation power. This asymmetry is manifested clearly in practice. For instance, on the mathematics construction task autocorrelation inequality 2, the strongest agent such as Claude-Code with Opus 4.6 plateaued at a score of 0.9438, even with hundreds of refinement steps at a cost of100million tokens and500USD. However, by effectively balancingCandL(method specified below), we match this score using the open-source gpt-oss-120b model ‡ at a cost of60USD (8.3gap). More surprisingly, we achieve a SOTA score of 0.9627 by continuously scaling up the evaluation (400USD), a score that no closed-source model could improve upon. This empirical evidence clearly demonstrates the importance of identifying the correct factor for evaluation scaling. Having demonstrated how compute should be allocated across multiple refinement trajectories, we turn to the scaling within each trajectory, with a given budgetL=N/C. At first glance, more refinement steps seem preferable, as each step can leverage feedback from previous attempts. However, this intuition could break down when path dependence is strong (βis large), when each refinement step further commits the trajectory to its current direction, making it harder to escape suboptimal regions. Local sample sizeK.The preceding analysis motivates a second budget axis: the local sample sizeK. While the number of independent trajectoriesCmitigates global path dependence by exploring different refinement paths,Kaddresses a more local failure mode: an individual refinement proposal may fail to improve the current solution. We model this by assuming that, after a dimension is selected according to the Pólya refinement rule in Definition 7.1, the generator producesKindependent refinement proposals for that dimension. Each proposal improves the selected dimension with probabilityp, and otherwise leaves it unchanged. The best proposal is then applied. Thus, increasingKraises the probability that a refinement step results in an actual improvement frompto1(1p) K , but it also reduces the number of sequential refinement steps under a fixed total budget. We empirically illustrate this trade-off in Figure 20. We simulate problems parameterized by(D,λ,β) under the refinement model in Theorem7.1, fixingL=4096,β=4, andC=32independent trajecto- ries, and report the normalized final bottleneck score averaged over 2048 simulations. The results show that moderate values ofKsubstantially improve performance over purely sequential refinement(K=1), especially when the improvement probabilitypis large enough for local sampling to reliably find a use- ful proposal. However, very largeKcan degrade performance, since allocating more budget to local sampling leaves fewer refinement steps. This confirms thatKshould be treated as a trade-off parameter rather than a monotonic source of improvement. We further study this interaction betweenC,T, andK on real open-ended tasks in Section 4.4. Together, the total evaluation budgetN=CKLhas been decomposed into three scaling dimensions: global independent trajectoriesC, local sample sizeK, and the actual refinement depthL= N CK . In short, this section aims to recognize the simple yet effective factors to be scaled, with the evidence below showing that by wisely scaling these parameters, we can achievestate-of-the-artsolutions on most problems consid- ered. Arguably, there are clear room for further improvements. For instance, current budget is divided evenly across each trajectory. Clearly, discarding unsatisfying trajectories on-the-fly and saving budgets for promising trajectories could be helpful. Additionally, one might recognize the fractal structure ofC andL, both being i.i.d. attempts at different levels. As an analogy, breakingLsequential refinement steps into smaller segments to couple more complexity might be beneficial. Relevant ablations can be found in Section 4.4. ‡ Estimated costs for gpt-oss-120b are calculated based on the median pricing from OpenRouter, which is $0.15 per 1M input tokens and $0.60 per 1M output tokens. Page 75 of110 Evaluation-driven Scaling for Scientific Discovery 1234816 Local Sample Size (K) 0.9 0.7 0.5 0.3 0.1 Improvement Probability (p) 0.150.500.330.250.120.06 0.130.450.330.250.120.06 0.080.270.290.240.120.06 0.050.100.140.180.120.06 0.020.020.020.020.040.05 Easy Problem (D=6) 1234816 Local Sample Size (K) 0.020.410.330.250.120.06 0.020.160.320.250.120.06 0.010.050.170.220.120.06 0.010.010.030.050.120.06 0.000.000.000.000.010.01 Hard Problem (D=12) 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Figure20:The score (higher the better) under Definition7.1under different refinement probabilitypand local sample sizeK. Herepis the probability that a proposal improves the selected dimension; otherwise it causes no change. ModerateKimproves performance, while overly largeKreduces the number of refinement steps and can hurt the final bottleneck score. CDiscovered Construction in Mathematics This appendix collects representative mathematical constructions discovered by SIMPLETES. We move these construction details out of the main text so the task descriptions can stay focused on the benchmark definitions, evaluation settings, and headline numbers. Erdős minimum overlap.The best witness is nearly binary, but not purely so: a small number of graded transition regions absorb the exact mass constraint while the bulk of the mass sits on long high / low plateaus. The solver first searches on a coarse grid, then refines the best candidates with constrained local improvement and projected polishing on finer discretizations. The governing principle is equioscillation rather than mere sparsity: mass is redistributed until the active translated overlaps become nearly tied, which is exactly the flattening visible in the right-hand panel. 0.00.51.01.52.0 x 0.0 0.2 0.4 0.6 0.8 1.0 h(x) Step witness 2.01.51.00.50.0 shift 0.0 0.2 0.4 0.6 0.8 1.0 relative max-overlap Normalized overlap profile Erds minimum overlap Figure 21:Best Erdős minimum-overlap witness produced by our evolved program. Left: the step-function witness on[0, 2]. Right: the normalized overlap profile. Autocorrelation inequalities.The three autocorrelation tasks reward visibly different witnesses and therefore different search parameterizations. AC1 uses a simplex-projected mass-transfer search on a1024- Page 76 of110 Evaluation-driven Scaling for Scientific Discovery bin non-negative witness; the final construction pushes mass toward the boundary and breaks the support into many short active blocks so the autoconvolution peak is spread across many shifts instead of concen- trating at a single one. AC2 is much more optimization-heavy: FFT-based convolutions and L-BFGS-B refinement on a smooth-max surrogate produce a sparse witness whose autoconvolution develops a long near-constant top plateau, matching the minimax character of the ratio objective. AC3 changes the search space again by optimizing a signed witness through a DCT-style parameterization, so cancellation is built into the variables themselves; the alternating positive and negative lobes lower the absolute peak while preserving unit mass. Sum-Difference Problem .This solver is the most explicitly combinatorial in the suite. It maintains exact sumset and difference-set multiplicity tables, alternates aggressive pruning with greedy insertions and local replacements, and quickly rejects edits that enlargeAAfaster thanA+A. The discovered set keeps a long arithmetic-progression backbone with gap4, then spends a very small number of irregular edits near the fringe, mostly+1and+3corrections with an occasional gap2, to create additional sums without paying the full difference-set cost. Since the construction itself is more informative than a stylized plot, we list both the discovered set and its consecutive-gap array directly below. Sum-Difference Problem construction SetA(506 integers; exact centered representative). [-487, -483, -482, -479, -478, -475, -474, -471, -470, -467, -466, -463, -462, -461, -459, -458, -455, -454, -451, -450, -447, -446, -443, -442, -439, -438, -435, -434, -430, -426, -422, -418, -414, -410, -406, -402, -398, -394, -390, -386, -382, -379, -378, -374, -370, -366, -362, -358, -354, -350, -346, -342, -338, -334, -330, -326, -323, -322, -318, -314, -310, -306, -302, -298, -294, -290, -286, -282, -278, -274, -271, -270, -266, -262, -258, -254, -250, -246, -242, -238, -234, -230, -226, -222, -218, -214, -211, -210, -206, -202, -198, -194, -190, -186, -182, -178, -174, -170, -166, -162, -158, -155, -154, -150, -146, -142, -138, -134, -130, -126, -122, -118, -114, -110, -106, -102, -99, -98, -94, -90, -86, -82, -78, -74, -70, -66, -62, -58, -54, -50, -46, -43, -42, -38, -34, -30, -26, -22, -18, -14, -10, -6, -2, 2, 6, 10, 13, 14, 18, 22, 26, 30, 34, 38, 42, 46, 50, 54, 58, 62, 65, 66, 70, 74, 78, 82, 86, 90, 94, 98, 102, 106, 110, 114, 118, 122, 125, 126, 130, 134, 138, 142, 146, 150, 154, 158, 162, 166, 170, 174, 177, 178, 182, 186, 190, 194, 198, 202, 206, 209, 210, 214, 218, 222, 226, 230, 234, 238, 242, 246, 250, 254, 258, 261, 262, 266, 270, 274, 278, 282, 286, 290, 294, 298, 302, 306, 310, 314, 318, 321, 322, 326, 330, 334, 338, 342, 346, 350, 354, 358, 362, 366, 370, 374, 377, 378, 382, 386, 390, 394, 398, 402, 406, 410, 414, 418, 422, 426, 429, 430, 434, 438, 442, 446, 450, 454, 458, 462, 466, 470, 474, 478, 482, 485, 486, 490, 494, 498, 502, 506, 510, 514, 518, 522, 526, 530, 534, 538, 542, 545, 546, 550, 554, 558, 562, 566, 570, 574, 578, 582, 586, 590, 594, 597, 598, 602, 606, 610, 614, 618, 622, 626, 629, 630, 634, 638, 642, 646, 650, 654, 658, 662, 666, 670, 674, 678, 681, 682, 686, 690, 694, 698, 702, 706, 710, 714, 718, 722, 726, 730, 734, 738, 741, 742, 746, 750, 754, 758, 762, 766, 770, 774, 778, 782, 786, 790, 793, 794, 798, 802, 806, 810, 814, 818, 822, 826, 830, 834, 838, 842, 846, 849, 850, 854, 858, 862, 866, 870, 874, 878, 882, 886, 890, 894, 898, 902, 905, 906, 910, 914, 918, 922, 926, 930, 934, 938, 942, 946, 950, 954, 958, 961, 962, 966, 970, 974, 978, 982, 986, 990, 994, 998, 1002, 1006, 1010, 1014, 1017, 1018, 1022, 1026, 1030, 1034, 1038, 1042, 1046, 1050, 1054, 1058, 1062, 1066, 1070, 1074, 1077, 1078, 1082, 1086, 1090, 1094, 1098, 1102, 1106, 1110, 1114, 1118, 1122, 1126, 1129, 1130, 1134, 1138, 1142, 1146, 1150, 1154, 1158, 1162, 1166, 1170, 1174, 1178, 1182, 1185, 1186, 1190, 1194, 1198, 1202, 1206, 1210, 1214, 1218, 1222, 1226, 1230, 1234, 1238, 1241, 1242, 1245, 1246, 1249, 1250, 1253, 1254, 1257, 1258, 1261, 1262, 1265, 1266, 1267, 1269, 1270, 1273, 1274, 1277, 1278, 1281, 1282, 1285, 1286, 1289, 1290, 1293, 1294, 1298] Gap array∆A(505 consecutive differences). Page 77 of110 Evaluation-driven Scaling for Scientific Discovery 0.20.10.00.10.2 x 0.000 0.005 0.010 0.015 0.020 0.025 height Witness 0.40.20.00.20.4 lag 0.0 0.2 0.4 0.6 0.8 1.0 normalized value Normalized autoconvolution First autocorrelation inequality (a)AC1 witness and normalized autoconvolution. 0.20.10.00.10.2 x 0.00 0.02 0.04 0.06 0.08 0.10 0.12 height Witness 0.40.20.00.20.4 lag 0.0 0.2 0.4 0.6 0.8 1.0 normalized value Normalized autoconvolution Second autocorrelation inequality (b)AC2 witness and normalized autoconvolution profile. 0.20.10.00.10.2 x 10 0 10 20 30 height Signed witness 0.40.20.00.20.4 lag 1000 500 0 500 1000 value envelope signed convolution Autoconvolution and envelope Third autocorrelation inequality (c)AC3 signed witness together with its autoconvolution envelope. Figure 22:Representative constructions for the autocorrelation inequality benchmarks. Page 78 of110 Evaluation-driven Scaling for Scientific Discovery [4, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 1, 2, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 1, 2, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 4] Circle packing in the unit square.The two circle-packing instances expose different geometric regimes. Forn=26, broad exploration over center layouts still matters: the final packing combines a dominant central circle, several boundary anchors, and a nearly triangular shell of medium circles, after which radii are re-optimized exactly against the induced contact pattern. Forn=32, the landscape is much stiffer. The successful search no longer invents a radically new topology; instead it stays close to an in- cumbent or six-row quasi-hexagonal template and extracts the remaining improvement from repeated linear-programming style radius optimization. The side-by-side figures show this contrast clearly: the n=26solution is hierarchical and heterogeneous, while then=32solution is denser, flatter, and much closer to a uniform contact pattern. Hadamard Maximum Determinant, order 29.The HM29 solver broadens the initialization rather than trusting a single classical construction: incumbents, quadratic-residue circulants, orthogonal-sign pat- terns, and cropped Sylvester matrices all seed the search. It then alternates inverse-guided multi-flip hill climbing with simulated annealing, allowing coordinated sign changes before the search cools into a sta- ble basin. We show the raw sign matrices directly, so the visible blocks match the actualf1gpatterns rather than a dephased representative. The baseline already exhibits the familiar structured seed, while the evolved matrix preserves the same determinant class with a more irregular local arrangement that still keeps row correlations tightly controlled. Page 79 of110 Evaluation-driven Scaling for Scientific Discovery Circle packing (n = 26) (a)n=26packing. Circle packing (n = 32) (b)n=32packing. Figure 23:Best circle-packing constructions produced by our evolved programs. column row Baseline order-29 matrix column row SimpleTES order-29 matrix HM29 maximum determinant Figure 24:HM29 comparison using raw sign matrices. Left: the baseline matrix provided in the literature, plotted with07!1. Right: the SimpleEvolve matrix loaded from our collected result. Page 80 of110 Evaluation-driven Scaling for Scientific Discovery DVariants of Proposal ConstructorΦ Balance policy.TheBalancepolicy is a stratified random sampling strategy designed to balance the exploitation of high-performing solutions with the exploration of diverse, sub-optimal trajectories. LetS be the set of historical attempts within a trajectory, sorted in descending order of their evaluator scores such thats 1 s 2 s jSj . To construct a proposal withninspirations, the policy enforces that the absolute best historical solution,s 1 , is always deterministically included. For the remainingn1slots, the policy categorizes the sorted historical trials into three overlapping tiers: an exploitation tierT exploit containing the topr elite fraction of attempts, an exploration tierT explore containing mid-tier solutions (typically between the 10th and 60th percentiles), and a global random tier T random =Scovering all prior attempts. Each subsequent inspiration is sampled without replacement according to the following probability distribution: Tier 8 > < > : T exploit with probabilityp exploit , T explore with probabilityp explore , T random with probability1p exploit p explore . (10) Once a tier is selected, an attempt is drawn uniformly at random from that tier. This heuristic ensures that the prompt is primarily grounded in elite solutions while consistently injecting varied structural contexts to prevent premature convergence. LLM-Elite policy.While score-based heuristics like the Balance policy are efficient, evaluating a trial’s potential solely by its scalar score can be myopic. TheLLM-Elitepolicy addresses this by leveraging semantic insights to dynamically maintain a bounded-size ”elite pool”P(with maximum capacityL elite ) that maximizes both solution quality and methodological diversity. Whenever a new candidate solutionx new is generated and evaluated, an auxiliary LLM acts as a gatekeeper. The LLM is provided with the scores and self-reflective summaries (detailing the approach and insights) of bothx new and all current solutions inP. It is instructed to output one of three actions: ADD, REPLACE(j) (to swap out a redundant or inferior attemptj2 P), or REJECT. To prevent the LLM from inadvertently discarding substantial progress due to misjudging diversity, we enforce a strict monotonic override rule: if the score ofx new is strictly greater than the maximum score currently inP, it bypasses the LLM’s rejection and is deterministically added (replacing the lowest-scoring solution ifjPj=L elite ). To sampleninspirations fromPfor a new proposal, we employ the identical stratified random sampling mechanism described in the Balance policy. The attempts inPare sorted in descending order by their scores and divided into the exploitation, exploration, and global random tiers. Trials are then sampled without replacement according to the previously defined tier probabilities. This approach heavily favors the inclusion of the highest-scoring elites while retaining a non-zero probability to draw inspiration from lower-ranked, yet semantically distinct, solutions curated by the LLM. Furthermore, this policy injects a concise text-only overview of the entire elite pool into the prompt (excluding raw code), allowing the generator to comprehend the global landscape of explored directions without consuming an excessive context window. Page 81 of110 Evaluation-driven Scaling for Scientific Discovery EPrompts This section presents the prompts utilized for each task in SIMPLETES. E.1Task instruction comparison. ThetaEvolve Lines 48 Words 361 Weighted parts 2 Guidance items 14 line count 48 word count 361 guidance items 14 Original prompt content SimpleTES Lines 4 Words 36 Task block 1 Guidance items 0 line count 4 word count 36 guidance items 0 Original prompt content (1) Part 1, weight=0.3: You are an expert mathematician specializing in circle packing problems and computational geometry. Your task is to improve a constructor function that directly produces a specific arrangement of core_parameters.n_circles circles in a unit square, maximizing the sum of their radii. The AlphaEvolve paper achieved a sum of target_value for n=core_parameters.n_circles The time limit for each program evaluation is MAX_RUNTIME seconds. Key geometric insights: - Circle packings often follow hexagonal patterns in the densest regions - Maximum density for infinite circle packing is pi/(2*sqrt(3)) approx 0.9069 - Edge effects make square container packing harder than infinite packing - Circles can be placed in layers or shells when confined to a square - Similar radii circles often form regular patterns, while varied radii allow better space utilization - Perfect symmetry may not yield the optimal packing due to edge effects Focus on designing an explicit constructor that places each circle in a specific position, rather than an iterative search algorithm. (2) Part 2, weight=0.7: You are an expert mathematician specializing in circle packing problems and computational geometry. We’re trying to reach the AlphaEvolve target of target_value for the sum of radii when packing core_parameters.n_circles circles in a unit square. The current implementation has plateaued at some values, so we need significant improvements. The time limit for each program evaluation is MAX_RUNTIME seconds. Key insights to explore: 1. The optimal arrangement likely involves variable-sized circles 2. A pure hexagonal arrangement may not be optimal due to edge effects 3. The densest known circle packings often use a hybrid approach 4. The optimization routine is critically important — simple physics-based models with carefully tuned parameters 5. Consider strategic placement of circles at square corners and edges 6. Adjusting the pattern to place larger circles at the center and smaller at the edges 7. The math literature suggests special arrangements for specific values of n 8. scipy has some useful functions for optimization Focus on breaking through the plateau by trying fundamentally different approaches — don’t just tune parameters. IMPORTANT: If you find the previous programs produce similar results, try as creative and evolutionary strategies as possible to explore different approaches. Pack 26 circles in a unit square to maximize the sum of their radii. Do this by evolving the code between # EVOLVE-BLOCK-START and # EVOLVE-BLOCK-END. The time limit for each program evaluation is 530 seconds. Figure 25:Comparison of prompt complexity between ThetaEvolve and SIMPLETES on the same Circle Packing in a Unit Square task. The verbatim original prompt text is preserved in each panel, while the summary metrics above each prompt quantify differences in length, structure, and the amount of explicit guidance. Page 82 of110 Evaluation-driven Scaling for Scientific Discovery E.2Erdős Minimum Overlap 1Find a step function h: [0, 2] -> [0, 1] that ** minimizes ** the overlap integral: 2 3$$C_5 = \\ max_k \ (x)(1 - h(x+k)) dx$$ 4 5** Constraints **: 61. h(x)in[0, 1]forallx 72. int_0 ^2 h(x) dx = 1 8 9** Discretization **: Represent hasn_points samples over [0, 2]. 10With dx = 2.0 / n_points: 11- 0 <= h[i] <= 1foralli 12-sum(h) * dx = 1 (equivalently:sum(h) == n_points / 2 exactly) 13 14Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 15 16The time limitforeach program evaluationis1100 seconds. Listing 6:Task instruction for Erdős Minimum Overlap E.3First Autocorrelation Inequality 1Discretize ([ -1/4 ,1/4]) into equal binsandsearchfora non -negative step function (f) that minimizes $ 2n (f*f)( )^2$, 2 3where (f*f) denotes the unnormalized discrete autoconvolution. 4 5Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 6 7The time limitforeach program evaluationis1100 seconds. Listing 7:Task instruction for First Autocorrelation Inequality E.4Second Autocorrelation Inequality 1Construct a non -negative function f on [-1/4, 1/4] to maximize 2 3R(f) = ||f * f||_2^2 / (||f * f||_1 * ||f * f||_inf),withC_2 >= R(f)andtarget R(f) > 0.97. 4 5Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 6 7The time limitforeach program evaluationis1100 seconds. Listing 8:Task instruction for Second Autocorrelation Inequality E.5Third Autocorrelation Inequality 1Design a Python program that constructs a discrete function `f: R -> R` on the domain `[-1/4, 1/4]` to minimize 2`C3 = 2 * n *max(|conv(f,f)|) / (sum(f))^2`, aiming to beat the SOTA of `1.4556427953745406 `. 3 4Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 5 6The time limitforeach program evaluationis70 seconds. Listing 9:Task instruction for Third Autocorrelation Inequality E.6Circle Packing in a Unit Square (n=26) 1Pack 26 circlesina unit square to maximize thesumof their radii. 2 3Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 4The time limitforeach program evaluationis530 seconds. Listing 10:Task instruction for Circle Packing in a Unit Square (n = 26) Page 83 of110 Evaluation-driven Scaling for Scientific Discovery E.7Circle Packing in a Unit Square (n=32) 1Pack 32 circlesina unit square to maximize thesumof their radii. 2 3Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 4The time limitforeach program evaluationis530 seconds. Listing 11:Task instruction for Circle Packing in a Unit Square (n = 32) E.8Hadamard Maximum Determinant, Order 29 1Find an n×n matrix Hwithelements equal to 1or-1 of size 29 to maximize |det(H)|. 2 3Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 4 5The time limitforeach program evaluationis350 seconds. Listing 12:Task instruction for Hadamard Maximum Determinant, Order 29 E.9Sum-Difference Problem 1In additive combinatorics ,fora finitesetof integers A, define 2 3- Sumset: A + A = a + b : a, binA 4- Differenceset: A - A = a - b : a, binA 5 6We want a strong lower boundforthe smallest constant C such thatforallfinite A subset of Z : 7 8|A + A| / |A| <= (|A - A| / |A|)^C 9 10For a candidatesetA, the evaluator computes 11 12C(A) = log(|A + A| / |A|) / log(|A - A| / |A|) 13 14andmaximizes C(A). 15 16Output requirements: 171. A must be a finitesetof integers. 182. 2 <= |A| <= 512 after deduplication. 193. Every element must bein[-1_000_000 , 1_000_000 ]. 20 21Program interface: 22- Implement run_code (). 23- run_code () mayreturneither: 24- (A, claimed_c), where Aisan iterable of integersandclaimed_cisafloat,or 25- A alone. 26 27The evaluator recomputes |A+A|, |A-A|,andC(A)fromA; reported values cannot be faked. 28The final rewardiscombined_score = C(A)forvalid outputs. 29 30Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 31 32The time limitforeach program evaluationis180 seconds. Listing 13:Task instruction for Sum-Difference Problem E.10Lasso Regularization Path 1You are an expertinnumerical optimizationandhigh -performance computingwithC++. 2Your taskisto write a fast Gaussian Lasso path solver that beats sklearn'simplementation. 3 4##Problem 5 6SolvethefullLassoregularizationpath: 7 8minimize(1/2n)||y-Xw||^2+lambda*||w||_1 9 10foradecreasingsequenceoflambdavalues(lambda_path),returningacoefficientmatrix Page 84 of110 Evaluation-driven Scaling for Scientific Discovery 11coef_pathofshape(p,n_lambda)wherecoef_path[:,k]arethecoefficientsatlambda_path[k]. 12 13Xisan(n,p)featurematrix,yisan(n,)targetvector,wisthe(p,)coefficientvector. 14 15##WhatYouProvide 16 17OnlytwothingsinsidetheEVOLVE-BLOCK: 18 191.**CPP_CODE**:ArawstringcontainingC++sourcecode.Theevaluatorwillcompile 20andrunthisasasubprocessthatreadsabinary-encodedproblemfromstdinand 21writesabinary-encodedcoef_pathtostdout. 22 232.**COMPILE_FLAGS**(optional):APythonlistofextracompilerflags,e.g. 24`COMPILE_FLAGS=["-fopenmp"]`ifyourcodeusesOpenMP. 25 26Theevaluatorhandlesallcompilation,subprocessinvocation,andtiming. 27YoudoNOTneedtowritealasso_path_solvefunction. 28 29##BinaryWireFormat 30 31Input(writtentobinarystdinbytheevaluator): 32int32n--numberofsamples 33int32p--numberoffeatures 34int32n_lambda--numberoflambdavalues 35float64[n*p]X(row-major) 36float64[n]y 37float64[n_lambda]lambda_path(decreasing) 38 39Output(writtentostdoutbyyourbinary): 40float64[p*n_lambda]coef_path(column-major:columnk=coefficientsatlambda[k]) 41 42##Evaluation 43 44**Timing**:EachproblemsizeisrunN_TIMING_RUNS=5times,eachwithadifferentrandom 45seed(differentX,y,lambda_path).Theminimumtimeisreported.Sinceeachcallreceives 46freshdata,cachingresultsacrosscallswillproducewronganswersandfailcorrectness. 47 48**Correctness**:Checkedonaseparatefreshproblemnotusedfortiming.Foreverylambda: 49 50obj(coef_yours[:,k])<=obj(coef_sklearn[:,k])+1e-6 51 52whereobj(w)=(1/2n)||y-Xw||^2+lambda*||w||_1.Allproblemsmustpassorthe 53overallscoreis0. 54 55**Score**:1/geo_mean(solve_time_ms).Higherisbetter. 56 57**Note**:Thetestcasesaresmallsizesforfastiteration.Yourprogrammustgeneralize 58toallproblemsizes,includingverylarge,high-dimensional,anddatasetswithdifferent 59levelsofsparsity. 60 61##BaselineAlgorithm:glmnet(AlreadyImplemented) 62 63ThecurrentinitsolverisafaithfulC++portofglmnet's Gaussian lasso path , 64implementing both of glmnet'smethodswiththesameswitchingrule(p<500uses 65thecovariancemethod,p>=500usesthenaivemethod).Bothmethodsincludewarm 66starts,sequentialstrongrulesforscreening,active-setinnerloop,andKKT 67checksonallpfeatures. 68 69Thestandardglmnetoptimizationsarealreadyinthebaseline.Reimplementing 70themwillnotimprovethescore.Tobeatthebaseline,youneedafundamentally 71differentalgorithmicapproach--notjustengineeringtweakstothesamecoordinate 72descentstructure.Thinkaboutwhatmakeslassopathsolvingslowandwhether 73therearesmarterwaystosolvethesamemathematicalproblem. 74 75**HARDCONSTRAINT--Precision**:YoursolverMUSTusefloat64(double)throughout 76allinternalcomputation,exactlymatchingthebaseline.Thisisnon-negotiable: 77-Allcoefficientvectors,residuals,gradients,andintermediatevaluesmustbedouble. 78-Switchingtofloat32internallyisforbidden,evenpartiallyortemporarily. 79-Programsthatusefloat32internallywillbedisqualifiedregardlessofcorrectnessscore. 80Thegoalisafasteralgorithmatthesameprecision,notalesspreciseone. 81 Page 85 of110 Evaluation-driven Scaling for Scientific Discovery 82##AvailableinC++ 83 84-Eigen3.xheaders(justuse`#include<Eigen/Dense>`) 85-FullC++17standardlibrary 86-Compilerflags:`-O3-march=native`(AVX2/AVX-512SIMDavailable) 87-OpenMP:set`COMPILE_FLAGS=["-fopenmp"]`ifusing`#pragmaompparallel` 88 89##Rules 90 91-Onlymodifycodebetween`#EVOLVE-BLOCK-START`and`#EVOLVE-BLOCK-END` 92-`CPP_CODE`mustbearawstringassignedtoavariablecalled`CPP_CODE` 93-`COMPILE_FLAGS`isoptional;ifpresentitmustbealistofstrings 94-DoNOTdefinea`lasso_path_solve`function--theevaluatorhandlesthis 95-NofilesystemornetworkIOinCPP_CODEoutsideofthestdin/stdoutwireformat 96 97##OutputStructure 98 99```python 100#EVOLVE-BLOCK-START 101 102CPP_CODE=r′ 103// Your C++ lasso path solver 104#include<Eigen/Dense> 105// ... 106intmain() 107// Read binaryinputfromstdin 108// Solve the lasso path 109// Write binary output to stdout 110 111′ 112 113#Optional:extracompilerflags 114#COMPILE_FLAGS=["-fopenmp"] 115 116#EVOLVE-BLOCK-END 117``` 118 119##Timebudget 120 121600secondsperevaluation(includingcompilation).Compilationiscachedacrosscalls 122withinthesameevaluationrunviacontent-hash,sorecompilationonlyhappenswhen 123CPP_CODEchanges. Listing 14:Task instruction for Lasso Regularization Path E.11Single-Cell RNA-Seq Denoising 1You are an expertincomputational biologyandsingle -cell RNA -seq analysis. 2Your taskisto develop a denoising algorithmforscRNA -seq count data. You are experiencedin 3computational biology librariesandtoolsandare familiarwithproblemsindenoisinginthe single -cell field. 4 5##Problem 6 7Single -cell RNA -seq dataisnoisy due to technical dropoutandlow capture efficiency. 8Given noisy count data , predict the true expression levels. 9 10Your predictionisevaluated against held -out molecules using two metrics: 111. **MSE** - Mean Squared Errorinlog -normalized space 122. ** Poisson Loss** - Poisson negative log -likelihood 13 14You need to implement a novel denoising algorithm that outperforms the current state -of -the -art without overfitting. 15 16##DataFormat 17 18- Input `X`: numpy array of shape (n_cells , n_genes) - **raw count data** 19- Output: numpy array of same shape - your denoised counts 20 21##Evaluation 22 Page 86 of110 Evaluation-driven Scaling for Scientific Discovery 23Your outputisevaluated using these exact functions: 24 25```python 26defevaluate_mse(test_data , denoised): 27test_X = scprep.utils.toarray(test_data).copy() 28denoised_X = np.asarray(denoised).copy() 29 30test_adata = anndata.AnnData(X=test_X) 31denoised_adata = anndata.AnnData(X=denoised_X) 32 33sc.p.normalize_total(test_adata , target_sum =10000) 34sc.p.log1p(test_adata) 35sc.p.normalize_total(denoised_adata , target_sum =10000) 36sc.p.log1p(denoised_adata) 37 38returnsklearn.metrics.mean_squared_error(test_adata.X, denoised_adata.X) 39``` 40 41```python 42defevaluate_poisson(train_data , test_data , denoised): 43test_X = scprep.utils.toarray(test_data) 44denoised_X = np.asarray(denoised).copy() 45 46initial_sum = train_data.sum() 47target_sum = test_X.sum() 48denoised_scaled = denoised_X * target_sum / initial_sum 49 50returnpoisson_nll_loss(test_X , denoised_scaled) 51``` 52 53##Scoring 54 55** Poissonisa HARD CONSTRAINT .** Your solutionisREJECTEDif`poisson_norm < 0.97 `. 56- `poisson_norm = (0.257575 - poisson) / (0.257575 - 0.031739) ` 57- MAGIC baseline achieves ~=0.97 58 59** Reward = MSE score only** (after passing Poisson constraint). 60 61##Budget&Resources 62 63- **Time budget **: 400sforyour code to run. You should time your codeandmake sure it runs within the time budget. 64- **CPUs **: 8 available 65 66##FunctionSignature 67 68```python 69defmagic_denoise(X, ** kwargs): 70#kwargsmayinclude:budget_s,random_state,knn,t,n_pca,solver,decay,knn_max,n_jobs 71#Youcanaddyourownparameterstoo 72#Yourimplementation 73returndenoised_X#sameshapeasX 74``` 75 76##Rules 77 78- Implement `magic_denoise(X, ...)` that returns denoised data 79- Use numpy , scipy , sklearn , graphtools , scprep , scanpy 80- Makeallhelper functions top level , no closuresorlambdas 81- No filesystemornetwork IO 82 83##KeyInsightsfromBenchmarks 84 85- NORMALIZATION ORDER MATTERS: Denoise raw/log counts first , then normalize."Reversed normalizationorder"achieves Poisson ~0.98 vs ~0.55forstandard order. 86- Square root transformisvariance -stabilizingforPoisson distributions 87- Poisson lossishighly affected by low non -zero values - push values < 1 toward zero 88- The original MAGICwithreversednormalization achieves best results 89 90##CRITICAL:Youroutputmusthavethisexactstructure 91 Page 87 of110 Evaluation-driven Scaling for Scientific Discovery 92```python 93#EVOLVE-BLOCK-START 94#<yourimportshere> 95#<yourhelperfunctionshere> 96 97defmagic_denoise(X, ** kwargs): 98#Yourimproveddenoisingimplementation 99returndenoised_X 100 101#EVOLVE-BLOCK-END 102``` 103 104- ONLY modify code between `#EVOLVE-BLOCK-START`and`#EVOLVE-BLOCK-END` 105- Copy everything after `#EVOLVE-BLOCK-END`EXACTLYasshownabove,unchanged 106 107##Timebudget 108 109The time limitforeach program evaluationis300 seconds. Listing 15:Task instruction for Single-Cell RNA-Seq Denoising E.12AHC039 – Purse Seine Fishing 1You are a world -classalgorithm engineer ,andyou are very good at programming. 2Now , you are participatingina programming contest. You are asked to solve a heuristic problem , knownasan NP -hard problem. Hereisthe problem statement: 3 4Story 5-------- 6Takahashiisa skilled purse seine fisher. 7His fishing boatisequippedwithstate -of -the -art sonar , allowing him to accurately determine the positions of fish within the fishing area. 8Additionally , the boatiscapable of high -speed movement , enabling him to assume that fish remain stationarywhilehe sets up the fishing net. 9 10The fishing method involves using the boat to deploy netsandform a closed polygon , capturing the fish within the enclosed area. 11To optimize efficiency , each edge of the polygon formed by the nets must be aligned either parallel to the east -westornorth -south direction. 12Furthermore , due to the limited length of the nets equipped on the boat , the polygon must be constructed within these constraints. 13 14The fishing area contains two types of fish: mackerelsandsardines. 15For resource conservation reasons , sardines are currently prohibitedfrombeing caughtinthis fishing area. 16Any sardines caughtinthe net must be released back into the sea. 17Because this processislabor -intensive , Takahashi should focus on maximizing the catch of mackerelwhileavoiding sardinesasmuchaspossible. 18 19 20Problem Statement 21-------- 22There are $N$ mackerelsand$N$ sardines on a two -dimensional plane. 23Construct a polygon that satisfies the following conditionsandmaximize the value obtained by subtracting the total number of sardines inside the polygonfromthe total number of mackerels inside it. 24Note thatanypoints lying on the edges of the polygon are considered to be inside the polygon. 25 26###Conditions 271. The number of verticesinthe polygon mustnotexceed $1000$ ,andthe total length of its edges mustnotexceed $4 × 10^5$. 282. The coordinates of each vertex $(x, y)$ must be integers satisfying $0 ≤ x, y ≤ 10^5$. 293. Each edge of the polygon must be parallel to either the $x$ -axisorthe $y$ -axis. 304. The polygon mustnotself -intersect: non -adjacent edges mustnotshareanypoints ,and adjacent edges must only meet at their endpoints. 31 32 33Scoring 34-------- 35Let $a$ be the total number of mackerels inside the polygonand$b$ be the total number of sardines inside the polygon. Page 88 of110 Evaluation-driven Scaling for Scientific Discovery 36Then , you will obtain the score of $ (0, a - b + 1)$. 37 38There are $150$ test cases ,andthe score of a submissionisthe total scoreforeach test case . 39If your submission produces an illegal outputorexceeds the time limitforsome test cases , the submission itself will be judgedasWAorTLE ,andthe score of the submission will be zero. 40The highest score obtained during the contest will determine the final ranking ,andthere will be no system test after the contest. 41If more than one participant gets the same score , they will be rankedinthe same place regardless of the submission time. 42 43 44Input 45-------- 46InputisgivenfromStandard Inputinthe followingformat: 47~~~ 48$N$ 49$x_0$ $y_0$ 50$ $ 51$x_2N-1$ $y_2N-1$ 52~~~ 53 54- Inalltest cases , the number of mackerelsandsardines , $N$ ,isfixed at $5000$. 55- For each $i = 0, 1, … , N-1$, $(x_i , y_i)$ represents the coordinates of the $i$ -th mackerel. 56- For each $i = 0, 1, … , N-1$, $(x_N+i, y_N+i)$ represents the coordinates of the $i$ - th sardine. 57- Each coordinate $(x_i , y_i)$ satisfies $0 ≤ x_i , y_i ≤ 10^5$,andallcoordinates are distinct. 58 59 60Output 61-------- 62Let the number of verticesinthe polygon be $m$ ($4 ≤ m ≤ 1000$),andlet $(a_i , b_i)$ denote the coordinates of the $i$ -th vertex. 63Then , output to Standard Outputinthe followingformat: 64~~~ 65$m$ 66$a_0$ $b_0$ 67$ $ 68$a_m-1$ $b_m-1$ 69~~~ 70 71The output vertices donotnecessarily need to form the actual corners of the polygon. 72In other words , three consecutive vertices $(a_i , b_i), (a_i+1, b_i+1), (a_i+2, b_i+2)$ may lie on a straight line. 73However ,allvertices must have distinct coordinates. 74 75The vertices can be outputineither clockwiseorcounterclockwise order. 76 77Your program may output multiple solutions. 78If multiple solutions are output , only the last oneisusedforscoring. 79 80================================================================================ 81CODE STRUCTURE 82================================================================================ 83 84The evolve block contains a single Python variable CPP_CODE which holds the 85entire C++ programasa triple -quoted string: 86 87#EVOLVE-BLOCK-START 88CPP_CODE =′ 89#include<iostream> 90... 91intmain() 92... 93return0; 94 95′ 96#EVOLVE-BLOCK-END Page 89 of110 Evaluation-driven Scaling for Scientific Discovery 97 98IMPORTANT: The CPP_CODE variable must contain the complete , compilable C++ program. 99DonotincludeanyPython logicinthe evolve block other than the CPP_CODE assignment. 100 101================================================================================ 102RULES 103================================================================================ 104 1051. ESCAPE SEQUENCES: Inside the Python triple -quoted string , you MUST write \ 106(double backslash n)forC++ newline characters. For example: 107std::cout << result <<"\ "; 108NOT: 109std::cout << result <<" "; 110Using a literal causes the tester to see onelongline instead of separate 111lines — this causes a Parse errorandscores zeroforALL 150 test cases. 112 1132. EVOLVE BLOCK MARKERS: You MUST include the exact markers: 114#EVOLVE-BLOCK-START 115CPP_CODE =′ 116...yourC++code... 117′ 118#EVOLVE-BLOCK-END 119 1203. COMPLETE CODE: The C++ program must be fully self -contained. If you reference 121a functionorvariable , it MUST be definedinthe code. Donotremove functions 122orglobalvariables that are used elsewhereinthe program. 123 1244. INCLUDES: Always use#include<bits/stdc++.h>orincludeALLnecessaryheaders. 125 1265. OUTPUT FORMAT: The program must output exactly: 127m 128a_0 b_0 129... 130a_m-1 b_m-1 131Every test case must produce valid output. If ANY of 150 test cases fails , 132the TOTAL scoreiszero. 133 1346. SEARCH REQUIRED: The program MUST implement a time -based search loop that 135actively improves the polygon until the time limitisreached. A static , 136greedy -only ,orhardcoded solution (suchasoutputting a fixed rectangle) 137isNOT acceptableandwill score near zero. The official time limitis2.0 seconds. 138The safety margin on time limit must be at least 0.05 seconds. 139 140================================================================================ 141PERFORMANCE TARGET 142================================================================================ 143 144Target average score per test case: 5000 145The best solution will make efficient use of the entire 2.0 second time limit 146without exceeding it. Think outside the box. Listing 16:Task instruction for AHC039 – Purse Seine Fishing E.13AHC058 – Apple Production Planning 1You are a world -classalgorithm engineer ,andyou are very good at programming. 2Now , you are participatingina programming contest. You are asked to solve a heuristic problem , knownasan NP -hard problem. You are trying to get the highest score possible to get the best rank on the leaderboard. Hereisthe problem statement: 3 4#Story 5 6APPLE ARTIS Corporation (commonly knownasAA Corporation)isa company engagedinthe mass production of apples. Recently , after many years of research , they have successfully developed an innovative machine capable of generating applesfromnothing. 7 8However , to begin full -scale mass production of apples using this machine , itisnecessary to mass -produce the machines themselves. To achieve this , A Corporation has established a hierarchical systeminwhich machines are created to produce apple -generating machines ,and machines are created to produce those machine -producing machines ,andso on. 9 Page 90 of110 Evaluation-driven Scaling for Scientific Discovery 10As an engineer at A Corporation , you have been taskedwithdeveloping a production planning algorithm that utilizes this hierarchy of machines to produceasmany applesaspossible. 11 12#ProblemStatement 13 14There are \(N × L\) types of machines , composed of \(N\) types of IDsand\(L\) types of Levels. A machinewithLevel \(i\)andID \(j\)isreferred toas** machine \(j^i\)** (\(0 ≤ i < L,\ 0 ≤ j < N\)). 15 16The production capacity of machine \(j^0\)is\(A_j\). The initial cost of machine \(j^i\)is \(C_i,j\). 17 18Your objectiveisto maximize the total number of apples at the end of \(T\) turns , following the procedure of the production plan below. 19 20##ProcedureoftheProductionPlan 21 22Let \(B_i,j\) be the number of machines \(j^i\),andinitiallyall\(B_i,j\) aresetto 1. 23Also , let \(P_i,j\) be the power of machine \(j^i\),andinitiallyall\(P_i,j\) aresetto 0. 24 25The initial number of apples at the start of the planis\(K\). 26Each turn proceeds according to the following steps: 27 281. You choose one of the following two actions: 29- Strengthen machine \(j^i\): Consume \(C_i,j × (P_i,j + 1)\) apples to increase \( P_i,j\) by 1. However , you cannot strengthenifit would resultina negative number of apples. 30- Do nothing. 312. Forallmachines \(j^i\), perform the followinginthe order of Level 0, 1, 2, 3: 32- For Level 0 machines (\(i = 0\)): 33- Increase the number of apples by \(A_j × B_i,j × P_i,j\). 34- For machines of Level 1orhigher (\(i ≥ 1\)): 35- Increase \(B_i-1,j\) by \(B_i,j × P_i,j\). 36 37Choose your actions wisely to maximize the number of apples at the end of \(T\) turns. 38 39#Scoring 40 41Let \(S\) be the number of apples at the end of \(T\) turns. Your scoreiscalculatedas\(\ mathrmround(10^5 × _2 S)\). 42The higher the score , the better. 43 44The following cases will resultina WA: 45 46- Performing a strengthening action that resultsinthe number of apples becoming less than \(0\) 47- Specifying a non -existent machine LevelorID 48- Taking fewer than \(T\) actions 49 50There are \(150\) test cases ,andthe score of a submissionisthe total scoreforeach test case. 51If your submission produces an illegal outputorexceeds the time limitforsome test cases , the submission itself will be judgedasWAorTLE ,andthe score of the submission will be zero. 52The highest score obtained during the contest will determine the final ranking ,andthere will be no system test after the contest. 53 54--- 55 56#Input 57 58InputisgivenfromStandard Inputinthe followingformat. 59 60``` 61N L T K 62A_0 A_1 ·s A_N-1 63C_0,0 C_0,1 ·s C_0,N-1 64C_1,0 C_1,1 ·s C_1,N-1 65 66C_L-1,0 C_L-1,1 ·s C_L-1,N-1 Page 91 of110 Evaluation-driven Scaling for Scientific Discovery 67``` 68 69- The first line contains four integers \(N, L, T, K\): 70- \(N\)isthe number of machine IDs ,and\(N = 10\). 71- \(L\)isthe number of machine Levels ,and\(L = 4\). 72- \(T\)isthe total number of turns ,and\(T = 500\). 73- \(K\)isthe number of apples at the start of the plan ,and\(K = 1\). 74- The second line contains \(N\) space -separated integers \(A_0 , A_1 , … , A_N -1\) representing the production capacities of Level 0 machines: 75- \(A_j\)isthe production capacity of machine \(j^0\), satisfying \(1 ≤ A_j ≤ 100\). 76- \(A\)issortedinascending order (\( A_0 ≤ A_1 ≤ ·s ≤ A_N -1\)). 77- The following \(L\) lines each contain \(N\) space -separated integers \(C_i,j\): 78- \(C_i,j\)isthe initial cost of machine \(j^i\), satisfying \(1 ≤ C_i,j ≤ 1.25 \ times 10^12\). 79 80#Output 81 82Output exactly \(T\) lines. 83Each line should describe the action taken on turn \(t\) (\(0 ≤ t < T\)),inorderfromturn 0, using the followingformat: 84 85- To strengthen machine \(j^i\): 86 87``` 88i j 89``` 90 91- To do nothing: 92 93``` 94-1 95``` 96 97Your program may include comment linesinthe output that startwith`#`. 98 99#InputGeneration 100 101The function \(\ mathrmrand\_double (L, U)\) represents generating a real number uniformly at random between \(L\)and\(U\). 102 103##Generationof\(A_j\) 104 105- When \(j = 0\):set\(A_0 = 1\) 106- When \(j ≠ 0\):set\(A_j = round(10^\ mathrmrand\_double (0 ,2))\) 107- After generatingallvalues , sort the array \(A\)inascending order 108 109##Generationof\(C_i,j\) 110 111- When \(i = 0\)and\(j = 0\):set\(C_0,0 = 1\) 112- Otherwise:set\(C_i,j = round(A_j × 500^i × 10^\ mathrmrand\_double (0 ,2))\) 113 114================================================================================ 115CODE STRUCTURE 116================================================================================ 117 118The evolve block contains a single Python variable CPP_CODE which holds the 119entire C++ programasa triple -quoted string: 120 121#EVOLVE-BLOCK-START 122CPP_CODE =′ 123#include<iostream> 124... 125intmain() 126... 127return0; 128 129′ 130#EVOLVE-BLOCK-END 131 132IMPORTANT: The CPP_CODE variable must contain the complete , compilable C++ program. Page 92 of110 Evaluation-driven Scaling for Scientific Discovery 133DonotincludeanyPython logicinthe evolve block other than the CPP_CODE assignment. 134 135================================================================================ 136RULES 137================================================================================ 138 1391. ESCAPE SEQUENCES: Inside the Python triple -quoted string , you MUST write \ 140(double backslash n)forC++ newline characters. For example: 141std::cout << result <<"\ "; 142NOT: 143std::cout << result <<" "; 144Using a literal causes the tester to see onelongline instead of 500 145separate lines — this causes a Parse errorandscores zeroforALL 150 test cases. 146 1472. EVOLVE BLOCK MARKERS: You MUST include the exact markers: 148#EVOLVE-BLOCK-START 149CPP_CODE =′ 150...yourC++code... 151′ 152#EVOLVE-BLOCK-END 153 154 1553. COMPLETE CODE: The C++ program must be fully self -contained. If you reference 156a functionorvariable , it MUST be definedinthe code. Donotremove functions 157orglobalvariables that are used elsewhereinthe program. 158 1594. INCLUDES: Always use#include<bits/stdc++.h>orincludeALLnecessaryheaders. 160 1615. OUTPUT FORMAT: Output exactly T=500 lines , each either"ij"or"-1". 162The program MUST NOT exit early underanycondition — no earlyreturn, no 163breakout of the main turn loop. Evenifno actionistaken , output"-1". 164Outputting fewer than 500 lines causes"Notenoughactions"WAandscores 165zeroforALL 150 test cases. 166 1676. SEARCH REQUIRED: The program MUST implement a time -based search loop that 168actively improves the solution until the time limitisreached. A one -pass 169greedy solution , however sophisticated ,isNOT acceptableandwill score 170significantly below the best possible. The time limitis2 seconds. 171 172 173================================================================================ 174PERFORMANCE TARGET 175================================================================================ 176 177Target average score per test case: 6 ,500 ,000 178The best solution will make efficient use of the entire time budget without 179exceeding it. Listing 17:Task instruction for AHC058 – Apple Production Planning E.14Batched Cumsum 1 2You are an expert Triton engineer taskedwithtranslating PyTorch code into highly optimized Triton kernel code. 3--- 41. Problem Definition 51.1 Task Description 6You will be implementing a batched inclusive cumulativesumkernel. The kernel computes the cumulativesumof a 2D float32 tensor along dim =1. 7 8Your taskisto implement this operation using Triton kernel codeandoptimize itforend -to - end latency on GPU. The implementation should preserve the same public interface. 9 10You may useanyprecisionformatandfully leverage the GPU hardware unitsforoptimization ,as longasoutput error stays below the required tolerance. (absolute tolerance: 0.0001 , relative tolerance: 0.0001) 11 121.2 Task Reference Implementation 13Hereisa pytorch implementation. You will want to implement a kernelforthe operationsinthe forward call: Page 93 of110 Evaluation-driven Scaling for Scientific Discovery 14 15#EVOLVE-BLOCK-START 16importtorch 17importtriton 18importtriton.languageastl 19fromtypingimportTuple 20 21defref_kernel(data: Tuple[torch.Tensor , torch.Tensor ]) -> torch.Tensor: 22x, output = data 23returntorch.cumsum(x, dim =1) 24 25#EVOLVE-BLOCK-END 26 271.3 Input/Output Specification 28| Tensor | Shape| Dtype | Layout| Role| 29|--------|--------------------|---------|------------|---------------| 30| x| [batch_size , n] | float32 | contiguous |input| 31| output | [batch_size , n] | float32 | contiguous | outputbuffer| 32 33`custom_kernel ` receives `(x, output)`, writes the inclusive cumsum along dim=1 into `output `, andreturns `output `. 34 35Theinputisalways 2D ([ batch_size , n]). The cumsumisalways along dim =1; each rowisan independent scan sequence. 36 37Typical shapes: 38- MoE dispatch: batch_size = num_experts –(128512) , n = num_tokens (32–K131K) 39- Nucleus sampling: batch_size = decode_batch –(3264) , n = vocab_size (201–K248K) 40 41--- 42 432. Hardware Specification 442.1 GPU Model 45NVIDIA H200 141GB 46 472.2 Key Hardware Parameters 48| Parameter| Value | 49|------------------------|----------| 50| SM Count| 132| 51| Max Threads per Block | 1024| 52| Registers per SM Count | 65536 | 53| Shared Memory per SM | 228 KB | 54| L2 Cache Size| 50 MB | 55| Memory Bandwidth| 4.8 TB/s | 56| FP32 Peak| 67 TFLOPS | 57| TF32 Tensor Core Peak | 989 TFLOPS | 58| FP16 Tensor Core Peak | 1,979 TFLOPS | 59| BF16 Tensor Core Peak | 1,979 TFLOPS | 60| FP8 Tensor Core Peak | 3,958 TFLOPS | 61 62--- 63 643. Implementation Guidelines 653.1 Triton version: 3.4.0 66 673.2 General Optimization Tips 68You are operatinginan iterative evolution loop ,nota one -shot rewrite setting. The objective isto produce reliable descendants that improve end -to -end geomean ,whilepreserving correctnessandstability. 69In this setting , each candidate should be treatedasa careful refinement of the strong known implementation ,nota fresh redesign. The goalisto improve end -to -end geomean , so each iteration should focus on a single optimization theme (forexample , launch tuning , local fusion , layout cleanup ,orreducing one materialization point). This keeps the search signal cleanandmakes it easier to identify which change actually improved performance. 70 71The optimization priority should be practical: first improve dataflow efficiency by removing unnecessary memory movement (extra reads/writes , transient tensors , avoidable layout conversions), then optimize compute -heavy regionswithcontext awareness. In particular , dominant contraction -like stages should be treated differentlyfromsurrounding elementwise andreduction code ,andproducer/consumer efficiency should usually be improved before replacing mature backend primitives. Page 94 of110 Evaluation-driven Scaling for Scientific Discovery 72 73To control regressions ,applychangeswitha risk —ladderlow -risk tuning first , medium -risk structural editsnext,andhigh -risk rewrites only when progress stalls. Throughout the process , preserve exact semantics acrossallrequiredinputregimesandmaintain numerically stable behaviorinreduction -sensitive paths. 74 753.3 A few general triton tips: 76- tl.arange only takesinconstexpr arguments (staticortl.constexpr) 77- You cannot usecontinueinyour kernel code 78- tl.dot can only takeintwoinputtensors 79- Thereisno tl.mean 80 81--- 82 834. Testing Requirements 84The following configurations provide example problem setups usedincorrectnessandperformance testing: 85 86Correctness: 87- "batch_size": 8,"n": 4096,"seed": 1717 88- "batch_size": 16,"n": 32000 ,"seed": 677 89- "batch_size": 32,"n": 64000 ,"seed": 324 90 91Performance: 92- "name":"a","batch_size":16,"n":32000 ,"seed":17717 93- "name":"b","batch_size":16,"n":262208 ,"seed":677 94- "name":"c","batch_size":32768 ,"n":32768 ,"seed":324 95- "name":"d","batch_size":64,"n":262208 ,"seed":23 96- "name":"e","batch_size":64,"n":32000 ,"seed":46 97- "name":"f","batch_size":96,"n":201088 ,"seed":46 98 99The scoreisthe reciprocal of the geometric mean acrossallbenchmark settings ,anda higher score indicates better performance. 100 101--- 102 1035. Output Format 104 105Your response must consist of oneandonly one complete , self -contained Pythonfile. Startwith `#EVOLVE-BLOCK-START`andendwith`#EVOLVE-BLOCK-END`. 106 107You can write custom_kernelandoptimized kernelasfollows: 108 109#EVOLVE-BLOCK-START 110importtorch 111importtriton 112importtriton.languageastl 113fromtypingimportTuple 114 115#YourTritonkernelcodehere 116 117defcustom_kernel(data: Tuple[torch.Tensor , torch.Tensor ]) -> torch.Tensor: 118x, output = data 119#Writeinclusivecumsumalongdim=1intooutputandreturnit 120returnoutput 121 122#EVOLVE-BLOCK-END 123 124--- 125 126Important: 127- Return oneandonly one complete Pythonfilebetween `#EVOLVE-BLOCK-START`and`#EVOLVE- BLOCK-END`. 128- Preserve `custom_kernel(data)` interface:inputis`(x, output)`,return`output `. 129- The cumsumisalways along dim =1. Inputisalways 2D. 130- Use Triton kernelsasthe primary optimization method. You shouldnotuse torch.compile. 131- Thisisiterative evolution: prefer incremental improvements over full rewrites. 132- Apply one main optimization theme per iteration; keep changes attributable. 133- Maintain numerically stable behavior — errors accumulate overlongscans. 134- Donotintroduce persistentglobalcache/state. Listing 18:Task instruction for Batched Cumsum Page 95 of110 Evaluation-driven Scaling for Scientific Discovery E.15Asymmetric Matrix Multiplication 1You are an expert Triton engineer taskedwithtranslating PyTorch code into highly optimized Triton kernel code. 2--- 31. Problem Definition 41.1 Task Description 5You will be implementing a asymmetric matrix multiplication kernel , a computation pattern that appearsinthe LLM. The computationisC = A @ B where Ais[m, k]andBis[k, n], producing C of shape [m, n]. 6 7Your taskisto implement this operation using Triton kernel codeandoptimize itforend -to - end latency on GPU. The implementation should preserve the same public interface. 8 9- You may useanyprecisionformatandfully leverage the GPU hardware unitsforoptimization , aslongasoutput error stays below the required tolerance. (absolute tolerance: 0.01, relative tolerance: 0.01) 10 111.2 Task Reference Implementation 12Hereisa pytorch implementation. You will want to implement a kernelforthe operationsinthe forward call: 13 14#EVOLVE-BLOCK-START 15importtorch 16importtriton 17importtriton.languageastl 18fromtypingimportTuple 19 20defref_kernel(data: Tuple[torch.Tensor , torch.Tensor , torch.Tensor ]) -> torch.Tensor: 21a, b, c = data 22returna @ b 23 24#EVOLVE-BLOCK-END 25 261.3 Input/Output Specification 27| Tensor | Shape | Dtype | Layout| Role| 28|--------|---------|---------|-----------------------|---------------| 29| a| [m, k] | float32 | contiguous , row -major |input| 30| b| [k, n] | float32 | contiguous , row -major |input| 31| c| [m, n] | float32 | contiguous| outputbuffer| 32 33`custom_kernel ` receives `(a, b, c)`, writes the matrix product a @ b into `c`,andreturns `c `. 34 35--- 36 372. Hardware Specification 382.1 GPU Model 39NVIDIA H200 141GB 40 412.2 Key Hardware Parameters 42| Parameter| Value | 43|------------------------|----------| 44| SM Count| 132| 45| Max Threads per Block | 1024| 46| Registers per SM Count | 65536 | 47| Shared Memory per SM | 228 KB | 48| L2 Cache Size| 50 MB | 49| Memory Bandwidth| 4.8 TB/s | 50| FP32 Peak| 67 TFLOPS | 51| TF32 Tensor Core Peak | 989 TFLOPS | 52| FP16 Tensor Core Peak | 1,979 TFLOPS | 53| BF16 Tensor Core Peak | 1,979 TFLOPS | 54| FP8 Tensor Core Peak | 3,958 TFLOPS | 55 56--- 57 583. Implementation Guidelines 593.1 Triton version 60Use Triton 3.4.0 style APIsandwrite kernels that are easy to autotune. 61 Page 96 of110 Evaluation-driven Scaling for Scientific Discovery 623.2 General Optimization Tips 63You are operatinginan iterative evolution loop ,nota one -shot rewrite setting. The objective isto produce reliable descendants that improve end -to -end geomean ,whilepreserving correctnessandstability. 64In this setting , each candidate should be treatedasa careful refinement of the strong known implementation ,nota fresh redesign. The goalisto improve end -to -end geomean , so each iteration should focus on a single optimization theme (forexample , launch tuning , local fusion , layout cleanup ,orreducing one materialization point). This keeps the search signal cleanandmakes it easier to identify which change actually improved performance. 65 66The optimization priority should be practical: first improve dataflow efficiency by removing unnecessary memory movement (extra reads/writes , transient tensors , avoidable layout conversions), then optimize compute -heavy regionswithcontext awareness. In particular , dominant contraction -like stages should be treated differentlyfromsurrounding elementwise andreduction code ,andproducer/consumer efficiency should usually be improved before replacing mature backend primitives. 67 68To control regressions ,applychangeswitha risk —ladderlow -risk tuning first , medium -risk structural editsnext,andhigh -risk rewrites only when progress stalls. Throughout the process , preserve exact semantics acrossallrequiredinputregimesandmaintain numerically stable behaviorinreduction -sensitive paths. 69 70 713.3 A few general triton tips: 72- tl.arange only takesinconstexpr arguments (staticortl.constexpr) 73- You cannot usecontinueinyour kernel code 74- tl.dot can only takeintwoinputtensors 75- Thereisno tl.mean 76- `num_stages ` pipeliningin`@triton.jit ` to overlapglobalmemory latencywithtensor core compute 77 78--- 79 804. Testing Requirements 81The following configurations provide example problem setups usedincorrectnessandperformance testing: 82 83Correctness: 84- "name":"a","m":16,"k":4096 ,"n":12288 ,"seed":17717 85- "name":"b","m":32,"k":11008 ,"n":4096 ,"seed":677 86 87Performance: 88- "m": 32,"k": 4096,"n": 12288 ,"seed": 677 89- "m": 16,"k": 11008 ,"n": 4096,"seed": 677 90- "m": 32678 ,"k": 16,"n": 32678 ,"seed": 324 91- "m": 32678 ,"k": 32,"n": 32678 ,"seed": 324 92- "m": 32678 ,"k": 64,"n": 32678 ,"seed": 324 93 94The scoreisthe reciprocal of the geometric mean acrossallbenchmark settings ,anda higher score indicates better performance. 95 96--- 97 985. Output Format 99 100Your response must consist of oneandonly one complete , self -contained Pythonfile. Startwith `#EVOLVE-BLOCK-START`andendwith`#EVOLVE-BLOCK-END`. 101 102You can write custom_kernelandoptimized kernelasfollows: 103 104#EVOLVE-BLOCK-START 105importtorch 106importtriton 107importtriton.languageastl 108fromtypingimportTuple 109 110#YourTritonkernelcodehere 111 112defcustom_kernel(data: Tuple[torch.Tensor , torch.Tensor , torch.Tensor ]) -> torch.Tensor: 113a, b, c = data 114#Writeresultintocandreturnc Page 97 of110 Evaluation-driven Scaling for Scientific Discovery 115#c[...]=your_triton_matmul(a,b) 116returnc 117 118#EVOLVE-BLOCK-END 119 120--- 121 122Important: 123- Return oneandonly one complete Pythonfilebetween `#EVOLVE-BLOCK-START`and`#EVOLVE- BLOCK-END`. 124- Preserve `custom_kernel(data)` interface:inputis`(a, b, c)`, write result into `c`,return `c`. 125- Use Triton kernelsasthe primary optimization method. You shouldnnotuse torch.compile. 126- Thisisiterative evolution: prefer incremental improvements over full rewrites. 127- Apply one main optimization theme per iteration; keep changes attributable. 128- Maintain numerically stable behaviorforreduction -sensitive computations. 129- Donotintroduce persistentglobalcache/state. Listing 19:Task instruction for Asymmetric Matrix Multiplication E.16TriMul 1You are an expert Triton engineer taskedwithtranslating PyTorch code into highly optimized Triton kernel code. 2 3You will be implementing a Triangle Multiplicative Update (TriMul) module thatisa core operation 4forAlphaFold3 , Chai , Protenix ,andother protein structure prediction modelsinBioML. 5 6The TriMul operator operates over a 4D tensor of shape [B, N, N, C]. 7 8Your task: 9- Implement the"outgoing"version of the TriMul operatorfromthe AlphaFold3 paper. 10- You willnothave to computeorstore gradientsforthis version. You will only need to implement the forwardpass. 11 12Your function should be definedas'custom_kernel'withthe following signature: 13Input: 14- `data `: Tuple of (input: torch.Tensor , weights: Dict[str, torch.Tensor], config: Dict) 15-input: Input tensor of shape [bs , seq_len , seq_len , dim] 16- mask: Mask tensor of shape [bs , seq_len , seq_len] 17- weights: Dictionary containing model weights 18- config: Dictionary containing model configuration parameters 19 20Output: 21- output: Processed tensor [bs , seq_len , seq_len , dim] 22 23** Problem Constraints :** 24- Bin1,2, Nin128 ,256 ,512 ,1024 , cin128 , c_zin128 ,384 ,768 25- Theinputdistribution will be sampledfroma standard Normal distribution ,ora heavy -tailed Cauchy distribution (gamma = 2). 26- There will either be no mask ,ora randomly sampled mask over the inputs. 27 28** Remarks .** So whyisthis problem so annoying? Because you have to choose whether to load / dealwitheither the channel dimensions c,c_z that the LayerNorms require (otherwise you have to do a synchronize to compute the statistics like mean / variance)orthe sequence dimension N. 29The sequence dimensionisparticularly annoying because it'squitelarge,butalsobecausewe computepair-wiseoperationsatthelastoperationthatsumoveranothersequencedimension (thisisN^3!). 30However,Ireallylikethiskernelbecauseitonlyconsistsof"simple"operations,andis reallyeasytounderstand.Itisatruetestof"fusions"thattorch.compile()doesn't do that well. 31 32Hereisa pytorch implementation of the TriMul module. You will want to implement a kernelfor the operationsinthe forward call: 33 34```python 35importtorch 36fromtorchimportnn , einsum 37importmath Page 98 of110 Evaluation-driven Scaling for Scientific Discovery 38 39#ReferencecodeinPyTorch 40classTriMul(n.Module): 41def__init__( 42self , 43dim:int, 44hidden_dim:int, 45): 46super().__init__ () 47 48self.norm = n.LayerNorm(dim) 49 50self.left_proj = n.Linear(dim , hidden_dim , bias=False) 51self.right_proj = n.Linear(dim , hidden_dim , bias=False) 52 53self.left_gate = n.Linear(dim , hidden_dim , bias=False) 54self.right_gate = n.Linear(dim , hidden_dim , bias=False) 55self.out_gate = n.Linear(dim , hidden_dim , bias=False) 56 57self.to_out_norm = n.LayerNorm(hidden_dim) 58self.to_out = n.Linear(hidden_dim , dim , bias=False) 59 60defforward(self , x: torch.Tensor , mask: torch.Tensor) -> torch.Tensor: 61""" 62x:[bs,seq_len,seq_len,dim] 63mask:[bs,seq_len,seq_len] 64 65Returns: 66output:[bs,seq_len,seq_len,dim] 67""" 68batch_size , seq_len , _, dim = x.shape 69 70x = self.norm(x) 71 72left = self.left_proj(x) 73right = self.right_proj(x) 74 75mask = mask.unsqueeze (-1) 76left = left * mask 77right = right * mask 78 79left_gate = self.left_gate(x).sigmoid () 80right_gate = self.right_gate(x).sigmoid () 81out_gate = self.out_gate(x).sigmoid () 82 83left = left * left_gate 84right = right * right_gate 85 86out = einsum('...ikd,...jkd->...ijd', left , right) 87#Thiseinsumisthesameasthefollowing: 88#out=torch.zeros(batch_size,seq_len,seq_len,dim,device=x.device) 89 90##Computeusingnestedloops 91#forbinrange(batch_size): 92#foriinrange(seq_len): 93#forjinrange(seq_len): 94##Computeeachoutputelement 95#forkinrange(seq_len): 96#out[b,i,j]+=left[b,i,k,:]*right[b,j,k,:] 97 98out = self.to_out_norm(out) 99out = out * out_gate 100returnself.to_out(out) 101``` 102 103Hereissome example skeleton code of the entrypoint function you will create: 104```python 105#EVOLVE-BLOCK-START 106#importpackages 107defcustom_kernel(data) 108input_tensor , mask , weights , config = data Page 99 of110 Evaluation-driven Scaling for Scientific Discovery 109dim , hidden_dim = config["dim"], config["hidden_dim"] 110 111#Accessthegivenweightsofthemodel 112norm_weight = weights["norm.weight"] 113norm_bias = weights["norm.bias"] 114left_proj_weight = weights["left_proj.weight"] 115right_proj_weight = weights["right_proj.weight"] 116left_gate_weight = weights["left_gate.weight"] 117right_gate_weight = weights["right_gate.weight"] 118out_gate_weight = weights["out_gate.weight"] 119to_out_norm_weight = weights["to_out_norm.weight"] 120to_out_norm_bias = weights["to_out_norm.bias"] 121to_out_weight = weights["to_out.weight"] 122 123#PerformTriMul 124 125returnout 126#EVOLVE-BLOCK-END 127``` 128 129Tohelpyou understand which triton version we are using , hereissome example triton codefor an unrelated task: 130```python 131importtriton 132importtriton.languageastl 133 134@triton.jit 135defmatmul_persistent_ws_kernel( 136a_ptr , b_ptr , c_ptr , M, N, K, 137stride_am , stride_ak , stride_bk , stride_bn , stride_cm , stride_cn , 138BLOCK_M: tl.constexpr , BLOCK_N: tl.constexpr , BLOCK_K: tl.constexpr , 139): 140pid = tl.program_id(axis =0)#async_task0,1,2 141num_pid_m = tl.cdiv(M, BLOCK_M)#async_task0,1,2 142num_pid_n = tl.cdiv(N, BLOCK_N)#async_task0,1,2 143pid_m = pid // num_pid_m#async_task0,1,2 144pid_n = pid % num_pid_n#async_task0,1,2 145offs_m_1 = pid_m * BLOCK_M + tl.arange(0, BLOCK_M // 2)#async_task0,1,2 146offs_m_2 = pid_m * BLOCK_M + tl.arange(BLOCK_M // 2, BLOCK_M)#async_task0,1,2 147offs_n = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_N)#async_task0,1,2 148offs_k = tl.arange(0, BLOCK_K)#async_task0 149a_ptrs_1 = a_ptr + (offs_m_1[:, None] * stride_am + offs_k[None , :] * stride_ak)# async_task0 150a_ptrs_2 = a_ptr + (offs_m_2[:, None] * stride_am + offs_k[None , :] * stride_ak)# async_task0 151b_ptrs = b_ptr + (offs_k[:, None] * stride_bk + offs_n[None , :] * stride_bn)#async_task0 152acc_1 = tl.zeros (( BLOCK_M // 2, BLOCK_N), dtype=tl.float32)#async_task1 153acc_1 = tl.zeros (( BLOCK_M // 2, BLOCK_N), dtype=tl.float32)#async_task2 154forkinrange(0, tl.cdiv(K, BLOCK_K)):#async_task0,1,2 155a_1 = tl.load(a_ptrs_1)#async_task0 156a_2 = tl.load(a_ptrs_2)#async_task0 157b = tl.load(b_ptrs)#async_task0 158acc_1 += tl.dot(a_1 , b)#async_task1 159acc_2 += tl.dot(a_2 , b)#async_task2 160a_ptrs_1 += BLOCK_K * stride_ak#async_task0 161a_ptrs_2 += BLOCK_K * stride_ak#async_task0 162b_ptrs += BLOCK_K * stride_bk#async_task0 163c_1 = acc_1.to(tl.float16)#async_task1 164c_2 = acc_2.to(tl.float16)#async_task2 165c_ptrs_1 = c_ptr_1 + stride_cm * offs_m_1[:, None] + stride_cn * offs_n[None , :]# async_task1 166c_ptrs_2 = c_ptr_2 + stride_cm * offs_m_2[:, None] + stride_cn * offs_n[None , :]# async_task2 167tl.store(c_ptrs_1 , c_1)#async_task1 168tl.store(c_ptrs_2 , c_2)#async_task2 169``` 170 171A few general triton tips: 172- tl.arange only takesinconstexpr arguments (staticortl.constexpr) 173- You cannot usecontinueinyour kernel code 174- tl.dot can only takeintwoinputtensors Page 100 of110 Evaluation-driven Scaling for Scientific Discovery 175- Thereisno tl.mean 176 177Here are the different configs that your kernel will be tested on ("nomask"sets whether there will be no mask ,ora randomly sampled mask over the inputs): 178 179Test Casesforcorrectnessandruntime (optimize runtimeforthese): 180- "seqlen": 256,"bs": 2,"dim": 128,"hidden_dim": 128,"nomask": True ,"distribution":" normal" 181- "seqlen": 768,"bs": 1,"dim": 128,"hidden_dim": 128,"nomask": True ,"distribution":" cauchy" 182- "seqlen": 256,"bs": 2,"dim": 384,"hidden_dim": 128,"nomask": False ,"distribution":" normal" 183- "seqlen": 512,"bs": 1,"dim": 128,"hidden_dim": 128,"nomask": True ,"distribution":" normal" 184- "seqlen": 1024,"bs": 1,"dim": 128,"hidden_dim": 128,"nomask": True ,"distribution":" cauchy" 185- "seqlen": 768,"bs": 1,"dim": 384,"hidden_dim": 128,"nomask": False ,"distribution":" normal" 186- "seqlen": 1024,"bs": 1,"dim": 384,"hidden_dim": 128,"nomask": True ,"distribution":" normal" 187 188IMPORTANT 189- Your response must include EVOLVE -BLOCK -STARTandEVOLVE -BLOCK -END markers Listing 20:Task instruction for TriMul E.17Qubit Routing on Superconducting Quantum Computer 1#TASK:ImproveaRustQubit-RoutingPolicyforQASM3Circuits 2 3You are optimizing a ** qubit routing policy **inRust. The initial policyisan implementation of Qiskit LightSABRE. A fixed routing engine handles circuit parsing , state tracking , control flow , disjoint topology ,andoutput construction. Your jobisto improve the ** decision -making policy **: (1) predict an initial mappingfromlogical qubit to physical qubit (2) select SWAP at every routing decisionwhilekeeping everything correctandfast. 4 5##0)Whatyouareallowedtoedit 6You will edit **ONLY** inside the marked region: 7* `// EVOLVE -BLOCK -START ` 8* `// EVOLVE -BLOCK -END ` 9The evaluator will: 101.compilethe Rust projectwithyour mutated code , 112. run routing on a benchmark suite of circuitsanddevice graphs , 123. score your results. 13Write a short summary of your reasoningasRust comments right after `// EVOLVE -BLOCK -START `. 14 15##1)Formalproblemstatement 16###Inputs 17You are given: 181. A ** quantum circuit ** (C) over logical qubits ($L=0,… ,n-1$). 192. A ** hardware topology ** ($G=(P,E)$),withphysical qubits ($P=0,… ,m-1$)andundirected edges ($E \(p,q) p,q \$), where a 2-qubit gate can be executed natively only on an edge. 20###Output 21You must produce a routed physical circuit (C\') thatisvalid under (G). Formally , routing consists of: 22* an initial injective mapping ($ _0: L → P$), 23* a sequence of SWAP layers ($S_t E_t=0^T-1$) that update the mapping over time , 24* a gate schedule that respects the circuit dependenciesandensures that each 2-qubit gate ($ ( _i ,\ ell_j)$)isexecuted only when ($ _t(\ ell_i) , _t(\ ell_j) $). 25###Goal 26The evaluator computes a score over a benchmark suite of different quantum circuitsand hardware topologies. Your optimization objective: 27* ** Maximize `combined_score `**. 28For each benchmark case `i`, let: 29* `w_i ` = the case weightfromsuite metadata , 30* `orig_i ` = inserted CNOT count (1 SWAP = 3 CNOTs) by the baseline routing strategy 31* `add_i ` = inserted CNOT count by your proposed strategy. 32*ifrouting succeeds: `add_i = 3 * the routed circuit 's inserted SWAP count ` 33*ifrouting fails /isinvalid: `add_i = orig_i ` (that case gets no credit) Page 101 of110 Evaluation-driven Scaling for Scientific Discovery 34The evaluator uses: 35* total score: `combined_score = sum_i w_i * (orig_i - add_i)` 36So higherisbetter. In practice , this means: 37* reducing inserted SWAPsisthe main driver of score , 38* failures are very expensive because they zero out that case 's improvement , 39The evaluator also reports `depth `, runtime , validity ,andother aggregate metrics. Those are useful signals when reasoning about tradeoffs , but `combined_score `isthe actual optimization target. 40If your policy fails tocompile, panics , times out ,orproduces invalid routing onany benchmark instance , it receives a large penalty / fails. 41 42##2)Topologycontextforthecurrentbenchmarksuite 43The current SABRE suite evaluates each circuit on three device graphs: `q20 `, `willow `,or` heron_fez `. You may exploit generic graph structure , but your policy **mustnot** hardcode logic these three topologies specifically. Later evaluation will include broader unseen topologies. 44###q20 45Qualitative picture: 46* a small sparse 20-qubit lattice , 47* roughly a 4x5 grid / ladder structure , 48*withrow links , column links ,anda few diagonal cross -braces , 49* short pathsandmoderate local connectivity. 50###willow 51Qualitative picture: 52* a large planar , locally connected , grid -like / brickwork topology , 53* many local alternativesforrouting , 54* wider interior regions than q20 , 55* congestionandpath diversity can matter much more than on small graphs. 56###heron_fez 57Qualitative picture: 58* a large sparse heavy -hex-like topology , 59* manylonghorizontal chainswithperiodic vertical connectors , 60* typically lower local branchingandfewer alternate short paths than willow , 61* routing may benefitfromrespecting narrow corridorsandavoiding local congestion traps. 62###Topology-awarestrategyguidance 63You are encouraged to infer structural properties of the current coupling graphanduse themin your policy. Useful generic signals may include: 64* degree / branching structure , 65* path diversityandlocal bottlenecks , 66* graph distance distribution , 67* whether the graph behaves more like a narrow corridor , a lattice ,ora broader mesh , 68* edge centrality / bridge -like connectors , 69* how much the active/front -layer qubits are clusteredorspread out. 70The exact runtime graphisavailable through `TopologyView ` methods suchas 71`edges ()`, `neighbors ()`, `distance ()`, `distances ()`,and 72`connected_components () `. Use the actual runtime graph instance rather than 73hardcoding benchmark -specific edge lists. 74Notice: 75* your policy should workfor** arbitrary ** topologies supplied by the evaluator , 76* do **not** hardcode special cases by topology name ,filename ,orexact qubit count , 77* do **not** assume future evaluation only uses the three graphs above , 78*ifyou adapt behavior to topology , compute graph featuresfromthe actual graph instance at runtime. 79 80 81##3)Hardconstraints(mustnotviolate) 82###A.Compile+interfacestability 83* Your code mustcompileasRust (stable toolchain). 84* Do **not** changeanyfunction signaturesortrait implementations that the engine expects. 85* Do **not** add heavy dependencies. (Prefer: no new crates .) 86* The evaluator may run multiple seeds/trials. If you use randomness , it must be derivedfrom the provided seed/RNGandremain reproducible. 87* Total compilation + evaluation should be within a reasonable time budget (target: 20 minutes on the full benchmark). 88###B.Correctnessinvariants 89Your policy mustnotbreakrouting correctness: 90* Never predict an illegal initial mapping (missing qubits , etc.),orselect an illegal SWAP ( must be an edgeinthe coupling graph). 91* Donotassume special initial states. 92* The 2-qubit gates must be executedinthe original order. An equivalent circuitwitha different order would be considered by the evaluatorasinvalid. Page 102 of110 Evaluation-driven Scaling for Scientific Discovery 93###C.Fixedinterfacereference 94Belowisthe fixed scaffold. It already defines the surrounding typesandtrait ,anddoes necessaryimport. Your evolve 95block should use them exactlyasnamed below. 96- Output only the code that should appear between EVOLVE -BLOCK -STARTandEVOLVE -BLOCK -END. 97- Donotoutput a fullfile. 98- The evaluator replaces the entire previous evolve blockwithyour returned evolve block. This isnota difforpatch. 99- If you keep only a modified fragment suchasone `impl `orone helper , every omitted item fromthe previous evolve blockisdeletedandthe program will usually fail tocompile. 100- Donotrepeatanyimport, trait , struct , enum , impl ,orfunction thatisalready definedin the scaffold below , whichisoutside the evolve blockandwill be used directlyin compilation. 101- Any helper you call must either be definedinthe scaffoldorbe defined by you inside the evolve block. 102####Policyhooks 103```rust 104pub trait Policy: Clone 105fn choose_best_initial_layout( 106&mut self , 107ctx: &InitialLayoutContext <'_>, 108rng: &mut RngState , 109) -> Result <Vec <usize >, RouterError >; 110 111fn choose_best_swap( 112&mut self , 113ctx: &SwapSelectionContext <'_>, 114rng: &mut RngState , 115) -> Option <(usize , usize) >; 116 117``` 118####Scaffold 119[Omitted because of the length. Please refer to the repoforthe full instruction] 120 121##4)Baseline:LightSABRE(what’yourestartingfrom) 122Your initial `candidate.rs `isan implementation of Qiskit LightSABRE , You can mutate it to achieve a better algorithm. 123###Coreheuristicskeleton 124Itisa greedy heuristic -search router that maintains a ** front layer ** of currently executable two -qubit gatesinthe dependency DAG , enumerates ** candidate SWAPs touching qubitsinthe front layer**,andchooses the best SWAP by a cost function based primarily on ** graph distances ** between mapped qubitsforthe front layer plus a ** lookaheadset**,withan optional ** decay ** factor to discourage repeatedly swapping the same qubits. 125There are several techniques to improve runtimeandquality: 126* efficient scoring (relative/delta evaluation rather than full recomputation), 127* multiple trials , 128* “release ”valve behavior to guarantee progress on hard cases , 129* broader support obligations suchasdisjoint coupling graphsandcontrol flow. 130Your policy can evolve far beyond “classic LightSABRE ”scoring ,aslongasyou stay inside the ’engines interface by providing correct `choose_best_initial_layout `and`choose_best_swap ` policy ,andfinish routingforthe full benchmark suite within the time limit of 1000s. 131Use your imagination to design novel algorithms. You can change the heuristic arbitrarily , using beam/tabu/MCTS/simmulated annealing , anything you want. Believeinyourself that your own policy might be far better. 132Analyze the feedbackfromprevious runs because they provide youwithgrounded information about the real performance of designed algorithms ,andyou can make modifications accordingly. 133 134##5)Submissionchecklist 135- `candidate.rs ` compiles without modifyinganyotherfile. 136- Your returned evolve blockisself -contained: every struct , helper , `impl Default `,and`impl Policy ` neededfromthe previous evolve blockisstill present unless you intentionally removedorreplaced itwitha valid alternative. 137- Policy still selects only legal swaps (edgesinthe coupling graph). 138- Deterministic given the seed. 139- Leaves control -flowanddisjoint -topology handling to the fixed engine (donotdisable them). Listing 21:Task instruction for Qubit Routing on Superconducting Quantum Computer E.18Compilation for Zoned Neutral Atom Quantum Architecture Page 103 of110 Evaluation-driven Scaling for Scientific Discovery 1 2##1.ProblemDefinition 3 4###Physicalsetting 5Neutral -atom quantum computers trap ** individual neutral atoms **invacuumwith** optical tweezers ** (focused laser beams)anduse themasqubits. Unlike fixed superconducting qubits , atoms can be ** shuttled ** during the program so that **who can interact ** changes over time. This document targets a ** Zoned Neutral -Atom Architecture (ZNAA)**: spaceis split into ** zones ** (notably storage vs entangling),andthe compiler must respect that geometry when scheduling gatesandmotion. 6 7###Hardwaremodel:SLMandAOD 8- **SLM (static lattice):** A ** fixed ** grid of —trapsthink of itasthe ** memory board **. ** Spatial light modulators ** imprint the pattern. In ZNAA it includes at least a ** Storage zone** (idle qubits , better coherence)andan ** Entangling zone** (where multi -qubit interactions are executed). Coordinates on these grids are what the ** placer ** assigns. 9- **AOD (mobile transport):** **Acousto -optic deflectors ** provide ** movable ** row/column tweezer —linesthink of a ** routing bus **. They **pick up** atomsfromSLM (** Open **), ** move** the line geometry (** Move **),and**drop** atoms onto SLM sites (** Close **). The ** router ** emits legal sequences of these ops so the machine state matches each target placement snapshot. 10 11###Gateexecutiononthismachine 12- **1Q gates :** Implementedas** local laser pulses ** on selected qubits at their current locations (subject to the machine API). 13- **2Q gates (Rydberg):** The machine applies a **globalentangling pulse ** over the entangling region: itis**not** a naive “one isolated pair at a ”time API. Any two atoms that are ** simultaneously **inthe entangling zoneand** close enough ** (within **`rydberg_radius `** andpairing rules) can ** interactinthe same pulse **. The compiler must therefore place atoms so the **set** of intended pairs matches the **global** 2Q layer ,withno stray unpaired candidate. 14 15###Compilerrole(whatyoubuild) 16Your jobisto design a ** compiler algorithm **withfour —components **`Scheduler `**, **` ReuseAnalyzer `**, **`Placer `**, **`Router —`** that turns a logical **`ZNAACircuit `** into a **solver -produced plan** whose stages embed **`ZNAAOperation `** streams. Concretely: ** schedule ** gates into parallel **1q** / **2q** stages; ** analyze ** optional ** reuse ** across adjacent 2Q layers; ** place ** full **(zone , x, y)** snapshots; ** route ** AOD motion between consecutive snapshots. Success means a ** strictly legal ** plan that preserves the circuit; good solutions also ** shorten ** wall -clock timeand**reduce** costly transport whileimproving ** fidelity**-related metrics. 17 18###Mandatoryrulesandscoring 19Treat this ’documents ** pipeline contracts **, ** placement schema**,and**physical -operation ** rulesas**hard** requirements. Violations fail the circuitforscoring: per -circuit score is**`-1.0`** (see scoring section elsewhereifpresent), even when the rest of the pipeline looks plausible. 20 21###Optimizationpriorities 22First achieve ** correctness ** onallbenchmark circuits (** maximize `success_over_total `**). Only then optimize ** runtime **and** fidelity**-cost: fewer unnecessary operations , shorter motion ,andfewer illegal shortcuts. Runtimeandfidelity **both** matter; keep placements androutes ** physically valid ** at every step. 23 24##2.Pipeline(scheduler+reuseanalyzer+placer+router) 25 26###Scheduler—`schedule(circuit)` 27####Inputformat 28- `circuit: ZNAACircuit ` — ordered logical gateswithqubit indices startingfrom0. 29####Outputformat 30- `list[ZNAAStage]` — each stageislogically typed **`1q`**or**`2q`** (aliases like `rydberg ` normalize to **`2q`**). 31 32###Reuseanalyzer—`analyze(stages)` 33####Inputformat 34- `stages:list[ZNAAStage]` — the scheduler output ,instage order. 35####Outputformat 36- `list[list[int]]` — **stage -aligned **with`stages `:foreach stage index , alistof logical qubit IDs markedfor** outgoing reuse ** toward thenext2Q-relevant context. **Only `2q` stages may use a non -empty reuse row .** 37- ** Length :** **`len(reuse_info) ==len(stages) `** — exactly **one** reuse row per scheduler Page 104 of110 Evaluation-driven Scaling for Scientific Discovery stage index `i` (`reuse_info[i]` belongs to `stages[i]`). 38 39###Placer—`place(stages,reuse_info)` 40####Inputformat 41- `stages:list[ZNAAStage]`and`reuse_info:list[list[int]]` — same lengthas`stages `, row `i` describes reuse flagsforstage `i`. 42####Outputformat 43- ** Primaryreturn** `placements_by_2q_stage:list[list[list[tuple]]]` — nested lists mean: 44- ** Outermostlist**: one entry per **non -empty `2q` stage ** (a stagewith`stage_type =="2q "`andat least one gate),in**the same order ** those stages appearin`stages `. Call that index `gi `; it lines upwiththe ’solvers `two_q_stage_indices[gi]`. 45- ** Length (outermost):** **`len(placements_by_2q_stage) ==` the number of non -empty `2q` stages **in`stages ` (sameas`len(two_q_stage_indices)`inthe solver). 46- ** Middlelist** (`group `): an ** ordered sequence of full placement snapshots **forthat 2Q stage. `group [0]`isthe layout **at the start of that 2Q stage ** (immediately before the ’stagesglobal2Q pulseinthe stitched plan). `group[1]`, `group[2]`, ... are further snapshots ** within the same 2Q stage **; the solver routes ** pairwise ** along the chain ` snap[k] -> snap[k+1]`. A minimal valid groupislength 1 (only the pre -2Q layout). 47- ** Innerlist** (`snap `): length ** number of logical qubits ** `n_q `, indexed by ** logical qubitid** `q` (0-based , contiguous indices `0..n_q -1`asused elsewhereinthe pipeline). `snap[q]`isone celltupleforqubit `q`. 48- **Eachtuple** `snap[q]`isexactly ** three ** scalar -like values `(zone , x, y) `: `zone `is `0` (storage)or`1` (entangling); `x`and`y` are **integer -valued ** coordinatesinthat ’zones grid. Together they fix one legal SLM cell per qubitforthat snapshot. 49- **`initial_placement ` (attribute on the placer instance ,notthereturnvalue of `place `)**: after `place (...) ` returns , the solver reads `placer.initial_placement ` — a single full snapshot `list[tuple]` of length `n_q `, same `(zone , x, y)`formatasabove ,with** every ** qubitin** storage ** (`zone == 0`). Thisisthe layout ** beforeany2Q-stage routing **; the first plan block maps qubitsfromit. Implementations mustsetthis field when computing placements. 50 51###Router—`route(placements)` 52####Inputformat 53- `placements:list[list[tuple]]` — each itemisone **full** qubit -indexed placement snapshot; consecutive pairs are routedinorder. The solver may invoke routing on **only two** adjacent snapshots at a time. 54####Outputformat 55- `list[list[ZNAAOperation ]]` — **one route segment ** (a Pythonlistof operations) per ** adjacent ** placement pairintheinputlist. 56- ** Length :**if`route `iscalled on a chain **`placements `** of length **`K`** (i.e. **`K`** full snapshots), thereturnmust satisfy **`len(returned_segments) == K - 1`**, segment `j` realizing the transition **` placements[j] -> placements[j+1] `**. (The usual solver callis **`K == 2`**, yielding **one** segment .) 57 58Pipeline -wide , downstream stages assume ** exact ** adherence to the ** length contracts ** spelled out aboveforreuse , placer ,androuter outputs , plus consistent **`n_q `** widthandvalid `(zone , x, y)` tuples;anymismatchisa hard failure ,nota warning. 59- ** Indexing convention :** logical ** qubit IDs are 0-based ** (alignedwiththe circuit / placement vectors). On SLMandAOD grids , an ** empty ** cellisrepresented by **` _EMPTY_QUBIT_CELL == -1`**,notby qubitid`0`. 60 61##3.RulesforPlacer 62###Whatthiscomponentdoes 63- The placer chooses ** where ** every logical qubit sits on the SLM at important snapshots: a full ** initial ** layout (allstorage) via `placer.initial_placement `, plus ** grouped ** placement sequencesforeach non -empty `2q` stage (`placements_by_2q_stage `). 64- Each snapshotisa **full -width ** vector: index `q`islogical qubit `q`, value `(zone , x, y) `isits —cell `zone=0` storage , `zone=1` —entanglingso the router knows the exact **from/to ** geometryforevery qubit when it builds `ZNAAOperation ` streams. 65- Within a 2Q group , multiple snapshots describe **intra -stage ** evolution (e.g. rearrange after some routing); the solver connects themwith** pairwise ** route callsinorder , so the placer effectively specifies the ** target geometry ** the router must realize. 66###Constraints 67- In each 2Q stage first placement , qubits involvedin2Q gates must beinentangling zone (` zone=1`). Non -involved qubits must stayinstorage zone (`zone=0`) to satisfy evaluator checksandreduceunnecessary exposure. 68- Each 2Q pair must satisfy ** entangling adjacency **aschecked by the evaluator (` _adjacent_entangling_columns `). For two qubits at placements `(zone , x, y)`with** integer ** `x, y`, ** adjacent ** meansallof: **`zone == 1` (entangling)forboth **; **same `x`**; **same `y // 2`** (same pair -row band after integer division);and**`|y_a - y_b| == 1`** ( column indices are immediate —neighborsthe precise condition implementedincode). A 2Q Page 105 of110 Evaluation-driven Scaling for Scientific Discovery pair thatisnotadjacent under this definition causes `PlacementValidationError `. 69- Placement tuples must stay validas`(zone ,x,y)`withinteger -like coordinatesandlegal zone ids. Brokentupleschema can fail solver/evaluator before route execution. 70- Reused qubits must be pinned correctly across grouped placementsandacross adjacent 2Q blocks. If a reused qubit moves when it should stay fixed , evaluator flags placement inconsistency. 71- After the last scheduled stage ends , **all** qubits must beinthe storage zone (`zone=0`). Practically: the final snapshot of the last non -empty `2q` stage group (or` initial_placement `ifthere are no `2q` stages) must place every logical qubitinstorage. 72###Howitinfluencestime/fidelity,andhowtoimprove 73- Placer quality directly controls router workload because route operations realize placement deltas. Better geometric continuity usually means fewerorshorter moves. 74- ** Distance still matters :** shorter SLM/AOD travel usually means **less move time**andoften simpler routes , so reducing displacement between consecutive snapshotsisa ** strong baseline ** goal. Donotignore —proximityonly trade extra distanceforparallelism when the routing winisclear. 75- Keeping non -active qubitsinstorage reduces time outside storage (`timeE `)andhelps decoherence -related fidelity terms. Entangling only whatisneededisboth validand beneficial. 76- Incorporate router movability signals (batch compatibility , likely non -crossing groups) into placement decisions. A placement thatisslightly farther but far more parallelizable can still be globally faster. 77- Minimize qubit movement events whenever possible: reducing movement usually reduces transfer pick/drop events , which improves fidelityandlowers total time cost. 78- Favor deterministic tie -breakrulesinassignment/matching. This stabilizes benchmark varianceandmakes iterative optimization more reliable. 79###Howtodebug 80- For `PlacementValidationError `, first inspect the failing 2Q stage first placement: involved qubitsinentangling , othersinstorage ,andpair adjacency. These are the most common root causes. 81- If errors mention reused qubits moved , compare previous 2Q block last placement vs current 2Q block first placementforthose qubits. Then check intra -group snapshots where reused qubits should remain fixed. 82- For `FinalStorageZoneError `, check the machine positions after the last stage: every qubit should report `zone='storage '`. On the placer side , ensure the last snapshot of the last non -empty `2q` group (or`initial_placement ` when no `2q` stage exists) has `zone=0`for allqubits ,andthat the routerisallowed to endwitha `Close ` that drops every carried qubit back onto empty storage cells. 83- If routing later fails but placement was recently changed , diff old/new placements to identify which qubits created impossibleorhigh -conflict routes. Debug placementand routing jointly ,notindependently. 84 85##4.EvolutionRules 86###Modifyonlybetweenevolvemarkers(**mandatory**) 87- **Non -negotiable :**if`#EVOLVE-BLOCK-START`/`#EVOLVE-BLOCK-END`markersexist,youonly needtomodifythecontentstrictlyinsidethespanbetweenthem.Youdonotneedtorepeat anycontentoutsidethemarkers,anditwillnotbedetectedbythesystem.Your submissionmustincludetheopeningandclosingmarkersthemselvesexactly;otherwiseit willfail. 88###Outputcompletecodeinsideeditablespan(**mandatory**) 89- The evolve block must be **self -containedandexecutable **: fullclassbodies , **all** helpers , **no** `...`, `#TODO`,“rest”omitted,orpartialdiffs.Anythingthat**cannot** beimportedandrun**aswritten**isan**automatic**failure.**Donot**assumethe readerwillmergeyouranswerwitholdercode—mentallyifitisnotspelledoutbetweenthe markers,itdoesnotexist. 90###Donothackruntimeorevaluatorassumptions(**mandatory**) 91- ** Strictly forbidden :** monkey -patchingorrebinding `ZNAAMachine `, `Solver `, hardware configs , timing flags (e.g. `skip_open_close_chain_time `), evaluator hooks ,oranyshortcut that falsifies physics , scores ,orchecks. **Zero tolerance**--such tactics are treatedas ** integrity violations **,notclever optimizations ,andyield** failed ** runsand **`-1.0`** scoring. 92###Keepcomponentclassandmethodnamesstable 93- Do **not** rename the four pipeline classes **`Scheduler `**, **`ReuseAnalyzer `**, **`Placer `**, **`Router `**,ortheir entry methods **`schedule `**, **`analyze `**, **`place `**, **` route `**. The evaluator expects these exact identifiers when wiring the pipeline; changing them usually causesimport/attribute failuresanda ** failed ** circuitwithper -circuit score **`-1.0`**. 94###Youcanimproveonecomponentperrun 95- Iterative , focused optimizationispreferred over broad rewrites because failures are easier to localizeandrevert. Improve one component at a time unless a coupled changeisclearly required. Page 106 of110 Evaluation-driven Scaling for Scientific Discovery 96###Randomizedalgorithms 97- If you use **any** stochasticorrandomized logic (e.g. `random `, `numpy.random `,orseeded shuffles),set**all** random number seeds to **`1`** so compilationis** reproducible ** andevaluator scores are stable across runs. 98###Timebudgetandalgorithmiccomplexity 99- The evaluator currently enforces a ** total runtime budget of 1800 secondsforthe whole evaluation run **. A separate per -circuit compilation timeout may be disabledinthe current configuration , so you shouldnotrely on an early killforpathological cases. Your scheduler / reuse / placer / router must therefore ** finish comfortably within theglobal budget ** on the benchmarkset--** avoid ** exponential blow -ups , brute force over the full configuration space ,orunstoppable loops. ** Prefer ** clearly ** polynomial -time** procedures (e.g. near -linearor\(O(n^3)\) matching on problem -size parameters)andkeep constant factors reasonable. 100 101##5.CurrentTask 102###Yourtaskistoimprovetheplacercomponentofthecompilationalgorithm,youshould provideaPlacerclasswithplacemethodbetweentheevolveblockmarkers,andyoushould notchangeanythingaboutscheduler,reuseanalyzerandrouter. 103 104##6.KeyInsights 105###Forplacer,thekeytohugeimproveistheconsiderationofconcurrencyinrouting,soyou mustfigureouthowtherouterpackqubitmovementsintoparallelrearrangementjob,and trytogiveamoreparallelizableplacementsinyourplaceralgorithm. 106###Youshouldconsidertheplacementsasawholeproblem,insteadofplacingqbuitstageby stage. 107- If you find it hard to give a placementswithhigh concurrency , you shouldtrymofidy the placements of previous stage. 108- After assigning positionsforevery qubitinevery stages , your algorithm should tune placementsinsome stage to get higher concurrencyandlower distance. 109- You cantryRTT: after placing the last stage , treat the final placements like an initial placement ,andredo the problem reversely , this procedure can be repeatedforseveral time. 110###Despitetheimportanceofconcurrency,distanceisalsoimportant,soyoushouldtryto placequbitscloserifthatwon'thurttheconcurrency.Youarehighlyrecommandedtouse thecalc_physical_coordinatemethodinAbstractPlacerclass,whichtakes(zone,r,c)as input,andoutputtherealphysicalposition(x,y).Moredetailsareprovidedbelow. 111###Treatreuseinfoashints,nothardconstraints.Reusemorequbitscanreducetransfer operationandleadtohigherfidelity,buttheremightbesometradeoff.Ifreuseisin conflictwithroutingconcurrency,youwillneedtoconsidercarefully. 112 113##7.Evaluatorfeedback 114- If your algorithm performs very well on some circuits but significantly worse on others , treat that imbalanceasa **high -priority diagnostic signal**,notasincidental noise. The evaluator mayreturna ** worst 2Q-stage window ** so you can investigate the root cause of the weak cases instead of only reading aggregate scores. 115- Placement snapshots are shownaszero -based qubit -labeled strings suchas**`q0: [0, 0, 0], q1: [0, 2, 0]`** so the coordinates can be tied back to individual qubits immediately. 116- The stage summary inside error snapshotsiscompactandpair -based ,forexample **`(q0 , q1) `**or**`(q0 , q1), (q2 , q3) `** instead of a verbose gatelist. 117- The reported stage indexisthe continuous **2Q-stage ordinal ** that ignores intervening 1Q stages , so it should line upwiththe 2Q-specific numbering usedinthe failure snapshot. 118- Donotignore this signal. Analyze why the poor circuits regress , identify the underlying bottleneck ,andimprove the weak cases directly before tuning anythingelse. 119 120##8.Hardwareconfiguration 121defconfig_test (): 122returnZNAAConfig( 123storage_shape =(100 , 100), 124entangling_shape =(7, 40), 125readout_shape =(20, 20), 126time_1q =0.625 , time_2q =0.360 , time_readout =500.0 , time_transfer =15.0 , 127fidelity_1q =0.9997 , fidelity_2q =0.995 , fidelity_readout =0.998 , fidelity_transfer =0.999 , fidelity_execution =0.995 , 128coherence_time_storage =1e8 , coherence_time_else =1.5e6 , aod_accelerate =0.00275 , 129distance_storage =(3.0 , 3.0), distance_entangle =(10.0 , 12.0) , distance_readout =(6.0 , 6.0), 130distance_interzone =10.0 , rydberg_radius =2.0, delta =2.0, name="config_testonly" 131)] Listing 22:Task instruction for Compilation on Zoned Neutral-Atom Quantum Architecture Page 107 of110 Evaluation-driven Scaling for Scientific Discovery E.19Parallel Scaling Law 1Discover a scaling law function that models the relationship between model parameter count , parallel size ,andlanguage modeling loss. Here weapply`parallel_size ` transformations to theinput, execute forward passes of the modelinparallel ,andaggregate the ` parallel_size ` outputs. We call this method parallel scaling. 2 3** IMPORTANT: The scaling law function must use no more than 4 parameters .** 4 5Focus on mathematical accuracy across different parallel configurations , cross -dataset generalization , parameter efficiency (simple forms that can be fittedwithlimited data), andnumerical/theoretical stability. 6 7**DATA CHARACTERISTICS ** 8- Features: [num_params , parallel_size] - 2Dinput 9- Labels: loss - scalar output 10- Groups:'pile'and'stack'datasets (18 samples each) 11- Parameterrange: 5.36e8 to 4.38e9 parameters (536M to 4.38B) 12- Parallel sizes: [1, 2, 4] copies 13- Lossrangeby group: 14-'pile': 1.7938 to 2.1113 (higher loss values) 15-'stack': 0.9906 to 1.1722 (lower loss values) 16- Key observation: Increasing parallel_size decreases loss 17- parallel_size =1: avg loss 1.9780 (pile), 1.0972 (stack) 18- parallel_size =2: avg loss 1.9480 (pile), 1.0767 (stack) 19- parallel_size =4: avg loss 1.9259 (pile), 1.0635 (stack) 20- Experimental setup: Augmentinputwithparallel_size copies ,passthrough LLM , aggregate responses 21 22The function signatures must remain: 23 24```python 25defscaling_law_func(data_points , params): 26#data_points:(N,2)arraywithcolumns[num_params,parallel_size] 27#num_params:Arrayofmodelparametercounts 28#parallel_size:Arrayofparallelcopiesforinputaugmentation 29#params:Arrayofupto4parameters 30#Returns:Predictedlossvalues 31 32deffit_scaling_law(data_points , loss_values): 33#data_points:(N,2)arraywithcolumns[num_params,parallel_size] 34#num_params:Arrayofmodelparametercounts 35#parallel_size:Arrayofparallelcopiesforinputaugmentation 36#loss_values:Arrayofcorrespondinglossvalues 37#Returns:Optimizedparameters(upto4parameters) 38``` 39 40Writeallimprovements between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-ENDmarkers. 41 42You arenotallowed to useinput-dependent featuresinscaling_law_func , e.g., median /min/ max/ etc. 43 44Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 45 46The time limitforeach program evaluationis600 seconds. Listing 23:Task instruction for Parallel Scaling Law E.20Domain Mixture Scaling Law 1Discover a scaling law function that models the relationship between domain mixture proportions andmulti -domain loss values across different model sizes. 2 3** IMPORTANT: The scaling law function must use no more than 35 parameters .** 4 5Focus on mathematical accuracy across different model sizes , cross -domain generalization , parameter efficiency (simple forms that can be fittedwithlimited data),andnumerical/ theoretical stability. 6 7**DATA CHARACTERISTICS ** 8- Features: Domain proportions (5 domains) - array of shape (n_mixtures , 5) Page 108 of110 Evaluation-driven Scaling for Scientific Discovery 9- Labels: Multi -domain losses (5 domains) - array of shape (n_mixtures , 5) 10- Dataset size: 80 training (20 per model size) 11- Model parameter sizes: 70M, 160M, 410M, 1B parameters (4 separate groups) 12- Domain proportions: Each row sums to 1.0 (mixture weights) 13- Loss ranges: Domain losses span 1.8 -4.2 cross -entropy loss 14- Mixture configurations: Systematic exploration of different domain weight combinations 15- Thisisa multi -output regression problemwithcorrelated domain performances 16 17The function signatures must remain: 18 19```python 20defscaling_law_func(data_points , params): 21#data_points:(N,5)arraywithdomainproportionsfor5domains 22#proportions:Arrayofdomainmixtureproportions 23#params:Arrayofupto35parameters 24#Returns:Predictedmulti-domainlossvalues(N,5) 25 26deffit_scaling_law(data_points , loss_values): 27#data_points:(N,5)arraywithdomainproportionsfor5domains 28#loss_values:Arrayofcorrespondingmulti-domainlosses(N,5) 29#Returns:Optimizedparameters(upto35parameters) 30``` 31 32Writeallimprovements between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-ENDmarkers. 33 34You arenotallowed to useinput-dependent featuresinscaling_law_func , e.g., median /min/ max/ etc. 35 36Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 37 38The time limitforeach program evaluationis600 seconds. Listing 24:Task instruction for Domain Mixture Scaling Law E.21Learning-Rate and Batch-Size Scaling Law 1Discover a scaling law function that models the relationship between learning rate , batch size , data size , model parametersandtraining loss. 2 3You are allowed to decide the number of parametersinthe scaling law function. 4 5Focus on mathematical accuracy across different hyperparameter scales , cross -configuration generalization , parameter efficiency (simple forms that can be fittedwithlimited data), andnumerical/theoretical stability. 6 7**DATA CHARACTERISTICS (2702 total data points):** 8- Features: [lr , bsz , data_size , non_embedding_param_size] - 4Dinput 9- Labels: lm_loss - scalar output 10- Dataset size: 2702 total 11- Learning raterange: 2.44e-4 to 2.21e-2 (logarithmically spaced) 12- Batch sizerange: 16 to 2048 (powers of 2) 13- Data sizerange: 2.0e9 to 1.0 e11 tokens (2B to 100B tokens) 14- Parameterrange: 6.00e7 to 1.07e9 (60M to 1.07B non -embedding parameters) 15- Lossrange: 2.1 to 3.7 cross -entropy loss 16- Comprehensive hyperparameter sweep covering learning rateandbatch size effects 17 18The function signatures must remain: 19 20```python 21defscaling_law_func(data_points , params): 22#data_points:(N,4)arraywithcolumns[lr,bsz,data_size,non_embedding_param_size] 23#lr:Arrayoflearningrates 24#bsz:Arrayofbatchsizes 25#data_size:Arrayofdatasizes 26#non_embedding_param_size:Arrayofnon-embeddingparametersizes 27#Returns:Predictedlmlossvalues 28 29deffit_scaling_law(data_points , loss_values): 30#data_points:(N,4)arraywithcolumns[lr,bsz,data_size,non_embedding_param_size] 31#loss_values:Arrayofcorrespondinglmlossvalues 32#Returns:Optimizedparameters Page 109 of110 Evaluation-driven Scaling for Scientific Discovery 33``` 34 35Writeallimprovements between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-ENDmarkers. 36 37You arenotallowed to useinput-dependent featuresinscaling_law_func , e.g., median /min/ max/ etc. 38 39Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 40 41The time limitforeach program evaluationis600 seconds. Listing 25:Task instruction for Learning-Rate and Batch-Size Scaling Law E.22U-Shaped Scaling Law 1Discover a scaling law function that models the relationship between compute (FLOPs)andLLM performance on easy questions , which exhibits a characteristic U-shapedordouble descent pattern. 2 3** IMPORTANT: The scaling law function must use no more than 6 parameters to capture the U- shaped scaling pattern .** 4 5Focus on mathematical accuracy across different benchmark tasks , cross -task generalization , parameter efficiency (simple forms that can be fittedwithlimited data),andnumerical/ theoretical stability. The U-shaped patterniscritical: performance initially worsenswith scale before improving again. 6 7**DATA CHARACTERISTICS :** 8- Features: [log_flops] - 1Dinputinlog10 scale 9- Labels: brier_score - scalar output (negative values , more negative = better) 10- Dataset size: 389fortrain 11- Log FLOPsrange: [-0.9, 2.9] approximately (log10 of FLOPsin1E21 units) 12 13The function signatures must remain: 14 15```python 16defscaling_law_func(data_points , params): 17#data_points:(N,1)arraywithcolumns[log_flops] 18#log_flops:Arrayoflog10(FLOPsin1E21units) 19#params:Arrayofupto6parameterstocaptureU-shapedpattern 20#Returns:Predictedbrier_scorevalues(negative) 21 22deffit_scaling_law(data_points , loss_values): 23#data_points:(N,1)arraywithcolumns[log_flops] 24#log_flops:Arrayoflog10(FLOPsin1E21units) 25#loss_values:Arrayofcorrespondingbrier_scorevalues 26#Returns:Optimizedparameters(upto6parameters) 27``` 28 29Writeallimprovements between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-ENDmarkers. 30 31You arenotallowed to useinput-dependent featuresinscaling_law_func , e.g., median /min/ max/ etc. 32 33Do this by evolving the code between#EVOLVE-BLOCK-STARTand#EVOLVE-BLOCK-END. 34 35The time limitforeach program evaluationis600 seconds. Listing 26:Task instruction for Easy-Question U-Shaped Scaling Law Page 110 of110