Paper deep dive
Evolution or Illusion? Rethinking Evaluation in LLM Evolutionary Search
Tal Oved, Roi Pony, Oshri Naparstek, Udi Barzelay
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/18/2026, 4:42:16 AM
Summary
This paper critiques the standard evaluation methodology for LLM-driven evolutionary search, which typically relies on a single seed and fixed iteration count. The authors demonstrate that optimal performance depends heavily on the trade-off between search width (number of seeds) and depth (iterations), and that strategy rankings can invert based on the budget allocation. They propose a rigorous protocol using a full grid of seeds and iterations to determine the optimal split and report the seeds-by-iterations frontier.
Entities (10)
Relation Signals (12)
OpenEvolve → isevaluatedon → ADRS-Bench
confidence 95% · We evaluate three evolutionary search strategies on five optimization tasks... The three strategies are OpenEvolve
EvoX → isevaluatedon → ADRS-Bench
confidence 95% · We evaluate three evolutionary search strategies on five optimization tasks... EvoX (co-evolving its selection rule)
AdaEvolve → isevaluatedon → ADRS-Bench
confidence 95% · We evaluate three evolutionary search strategies on five optimization tasks... AdaEvolve (adaptive variant)
PRISM → ispartof → ADRS-Bench
confidence 92% · ADRS-Bench collects systems-optimization tasks... We use three ADRS-Bench system tasks... PRISM
Cloudcast → ispartof → ADRS-Bench
confidence 92% · ADRS-Bench collects systems-optimization tasks... We use three ADRS-Bench system tasks... Cloudcast
Transaction Scheduling → ispartof → ADRS-Bench
confidence 92% · ADRS-Bench collects systems-optimization tasks... We use three ADRS-Bench system tasks... transaction scheduling
SkyDiscover → implements → AdaEvolve
confidence 88% · All experiments run on SkyDiscover... using its strategy implementations
SkyDiscover → implements → OpenEvolve
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM-driven evolutionary search finds programs by launching seeds and iterating each one. Papers report a single budget setting, usually one seed run for a fixed number of iterations, and rank methods from that one point. We show this is not enough. We evaluate three evolutionary search strategies on five optimization tasks, commonly used by papers in the genre to report results. We run the analysis over a full grid of seeds and iterations. Our findings suggest that the best way to split a fixed budget between more seeds (width) and more iterations (depth) changes with the strategy, the task, and the total budget. Furthermore, we observe that the ranking of strategies also changes with the budget. On one task the strategy that looks worst at one seed is best at forty seeds. On another the best number of iterations is well below the value common in practice, so extra depth wastes budget that more seeds would turn into score. We provide a measurement protocol that reports the seeds-by-iterations frontier and practical guidance for using it.
Tags
Links
- Source: https://arxiv.org/abs/2609.19799v1
- Canonical: https://arxiv.org/abs/2609.19799v1
Trouble viewing inline? Open PDF directly →
Full Text
31,715 characters extracted from source content.
Expand or collapse full text
Evolution or Illusion? Rethinking Evaluation in LLM Evolutionary Search Tal Oved * Roi Pony Oshri Naparstek Udi Barzelay IBM Research Abstract LLM-driven evolutionary search finds pro- grams by launching seeds and iterating each one. Papers report a single budget setting, usu- ally one seed run for a fixed number of itera- tions, and rank methods from that one point. We show this is not enough. We evaluate three evolutionary search strategies on five optimiza- tion tasks, commonly used by papers in the genre to report results. We run the analysis over a full grid of seeds and iterations. Our find- ings suggest that the best way to split a fixed budget between more seeds (width) and more iterations (depth) changes with the strategy, the task, and the total budget. Furthermore, we ob- serve that the ranking of strategies also changes with the budget. On one task the strategy that looks worst at one seed is best at forty seeds. On another the best number of iterations is well below the value common in practice, so extra depth wastes budget that more seeds would turn into score. We provide a measurement protocol that reports the seeds-by-iterations frontier and practical guidance for using it. 1 Introduction Recent years have seen the emergence of a new paradigm for algorithm discovery, in which large language models generate and iteratively refine can- didate algorithms through evolutionary search (Ma et al., 2024; Fernando et al., 2023; Guo et al., 2024; Madaan et al., 2023). FunSearch found new math- ematical constructions this way (Romera-Paredes et al., 2024), AlphaEvolve extended the idea to algorithms and hardware kernels (Novikov et al., 2025), and open engines such as OpenEvolve repro- duce the loop for public use (Sharma, 2025). The execution of these methods is determined by two dimensions, width and depth. Width is the number of independent seeds launched, and the best result * Corresponding author: Tal.Oved@ibm.com Rethinking evaluation in LLM evolutionary search: Fixed Budget (퐵=푘⋅푡) Width( 푘seeds) Depth( 푡iterations) The Problem: HiddenPerformanceLandscapes The Impact: Ranking Inversion The Tradeoff Common Assesment Point Optimal Split Equal Budget Curve Figure 1: An illustration of the need for rigorous evalu- ation methodology in LLM-based evolutionary search. over seeds is kept. Depth is the number of itera- tions each seed runs. Every seed and every iteration spends LLM calls, so a study has a proposal budget B = kseeds× titerations. A method is usually run for one seed, sometimes up to three, for a fixed number of iterations, and methods are ranked from that single point. The budget is chosen by habit. This paper shows that a single point is not enough to evaluate or compare these methods (Henderson et al., 2018). In this work, we address the methodological gap by evaluating three evolutionary search strategies 1 arXiv:2609.19799v1 [cs.CL] 17 Sep 2026 on five optimization tasks over a grid of forty seeds by two hundred iterations. We find that the optimal width-versus-depth split depends on the strategy, task, and budget; that the strategy ranking changes with the budget, so on one task the strategy worst at one seed is best at forty; and that the best number of iterations is often well below common practice, wasting budget more seeds would turn into score. The message is practical. Report the seeds-by- iterations frontier, not one point. Run several seeds. Tune depth per strategy and task. Contributions. •A budget-grid evaluation protocol and practi- cal reporting guidance: replay logged trajecto- ries and compute the expected best score per split(k, t)with exact order statistics (§2, §5). •Evidence across three strategies and five tasks that the optimal width-versus-depth split de- pends on strategy, task, and budget (§4). •A ranking inversion: single-seed evaluation reorders the strategies relative to a multi-seed budget, so the common protocol can name the wrong winner (§4). Related work. LLM evolutionary search pairs an LLM proposer with an evaluator in an evolution- ary loop: FunSearch (Romera-Paredes et al., 2024), AlphaEvolve (Novikov et al., 2025), OpenEvolve (Sharma, 2025), GEPA (Agrawal et al., 2026), heuristic-design systems (Liu et al., 2024; Ye et al., 2024), sample-efficient variants (Lange et al., 2026), and recent work on harness engineering (Ishibashi et al., 2026). ADRS-Bench collects systems-optimization tasks for this setting (Cheng et al., 2025). Across these methods the budget is fixed in advance and runs are reported singly or in a handful. A parallel line studies how to spend a fixed sampling or search budget (Gideoni et al., 2026; Ellis and Castro, 2026; Brown et al., 2024; Schaeffer et al., 2025; Kazdan et al., 2025; Snell et al., 2024; Inoue et al., 2026), none comparing evolutionary strategies across a seeds-by-iterations budget as we do. Finally, because individual runs are noisy, a best-of-kscore inherently scales with k(Smith and Winkler, 2006). We connect these threads: we measure the width-versus-depth split for each strategy and task, and show the reported winner depends on it. 2 Method Setup. A strategy runs independent seeds. Seed iproduces a trajectory of candidate scores f(x i,1 ), . . . , f(x i,t ), wherefis the task objective (combined_score, higher is better). Its value at depth t is the running best M i (t) = max 1≤s≤t f(x i,s ).(1) Launching k seeds and keeping the best gives B k (t) = max 1≤i≤k M i (t),(2) and the budget is B = k t. Budget grid.From the logged trajectories of the n=40seeds we compute, for every split(k, t), the expected best scoreE[B k (t)]over random size-k subsets of the40observed seeds (a finite-sample expectation). We use exact order statistics on the per-seed valuesM i (t), so there is no resampling noise. This yields a score surface over seeds and iterations for each strategy and task (Fig. 2). We examine three axes: •Width marginal:E[B k (T)]againstkat full depth T=200, the value of more seeds. •Depth marginal:E[B 1 (t)]againsttfor one seed, the value of more iterations. •Frontier: for each total budgetB, the best expected score over all splits withk t ≤ B, and the split(k, t)that reaches it. That split is the recommended allocation at budget B. Replaying logged trajectories ensures exact com- parisons without requiring redundant runs. Ranking. At a budget point we rank strategies by expected best score. The single-seed proto- col common in practice is the pointk=1, t=T. We compare its ranking to the multi-seed budget k=40, t=T . Ranking reversion.We also ask how often a par- tial budget names the wrong order. We bootstrap the seeds: for each of two thousand resamples we rank the strategies by best-of-seeds score at a bud- get point and at the full budget, and record whether the orders differ. The reversion probability is the share of resamples that differ, zero at the full bud- get by construction. We read it over iterations at full width, seeds at full depth, and total budget at its best split. 2 125102040 10 20 50 100 200 EvoX iterations t (log) PRISM · EvoX 80 200 500 1000 2000 4000 125102040 10 20 50 100 200 Cloudcast · EvoX 80 200 500 1000 2000 4000 125102040 10 20 50 100 200 Txn scheduling · EvoX 80 200 500 1000 2000 4000 125102040 10 20 50 100 200 OpenEvolve iterations t (log) PRISM · OpenEvolve 80 200 500 1000 2000 4000 125102040 10 20 50 100 200 Cloudcast · OpenEvolve 80 200 500 1000 2000 4000 125102040 10 20 50 100 200 Txn scheduling · OpenEvolve 80 200 500 1000 2000 4000 125102040 seeds k (log) 10 20 50 100 200 AdaEvolve iterations t (log) PRISM · AdaEvolve 80 200 500 1000 2000 4000 125102040 seeds k (log) 10 20 50 100 200 Cloudcast · AdaEvolve 80 200 500 1000 2000 4000 125102040 seeds k (log) 10 20 50 100 200 Txn scheduling · AdaEvolve 80 200 500 1000 2000 4000 26.235 26.240 26.245 26.250 26.255 mean best combined_score 26.246 26.248 26.250 26.252 26.254 mean best combined_score 26.3 26.4 26.5 26.6 26.7 mean best combined_score 1.50 1.60 1.70 1.80 mean best combined_score ×10 − 3 1.35 1.38 1.40 1.43 1.45 1.47 1.50 mean best combined_score ×10 − 3 1.53 1.54 1.55 1.56 1.57 1.58 1.59 mean best combined_score ×10 − 3 4225 4250 4275 4300 4325 4350 4375 mean best combined_score 4300 4400 4500 4600 4700 4800 4900 mean best combined_score 5000 7500 10000 12500 15000 17500 20000 mean best combined_score Best score over the seeds x iterations grid Figure 2: Expected best combined_score over the seeds (k) by iterations (t) grid (EvoX top, OpenEvolve middle, AdaEvolve bottom). Log-scaled axes make the white iso-budget lines (k·t) straight. Stars mark the score-maximizing cells. The star sits in a different place in each panel: the best split depends on the strategy and the task, and the best depth is often well short of the full two hundred iterations. 3 Experimental Setup Engine and strategies. All runs use the ADRS engine (Cheng et al., 2025) withgpt-5-mini, keeping the model, objective, and evaluator identi- cal so only the strategy varies. The three strategies are OpenEvolve (islands plus archive, population 40; Sharma, 2025), EvoX (co-evolving its selection rule, Liu et al., 2026a), and AdaEvolve (adaptive variant, Cemri et al., 2026). Following prior work, we budget by proposals rather than raw LLM calls, using an 8,000-proposal limit (40× 200) to ensure an equal comparison. Tasks and objective.We use three ADRS-Bench system tasks (Cheng et al., 2025): PRISM (LLM-serving scheduler, measured by goodput); Cloudcast (broadcast planner, transfer cost); and transaction scheduling (throughput).We op- timize their nativecombined_scores (PRISM reciprocal(max_kvpr) + success_rate, transaction scheduling10 6 /(1 + makespan), Cloudcast cost- based), gating out invalid or reward-hacking can- didates via validity checks for comparability. But the transaction-scheduling evaluator is incomplete: it does not verify that a schedule covers the whole workload, so a partial-schedule exploit passes (Ap- pendix D); we keep whatever it accepts, as prior work does. PRISM and Cloudcast are determinis- tic; transaction scheduling draws one random work- load per evaluation, so we re-evaluate each seed’s best program to remove single-draw noise. We also analyze two math tasks (Appendix C): Circle Packing (Square), and Heilbronn (triangle). 4 Results The optimal split depends on strategy, task, and budget. Figure 2 shows the seeds-by-iterations surface for each strategy and task, with iso-budget lines and the best cell marked. The best cell sits in different places across panels. Table 1 reports the optimal split at full and 10% budgets, high- lighting two patterns. First, allocations diverge by strategy: on PRISM at 10% budget, EvoX is best deep and narrow (four seeds,196iterations) while OpenEvolve is best wide and shallow (36,22). Sec- ond, depth saturates early. AdaEvolve on transac- tion scheduling peaks at40iterations and EvoX on PRISM at60; further depth wastes proposals better spent on seeds. Consequently, the optimal allocation axis is highly strategy- and task-specific. 3 TaskStrategyFull (k, t) 10% (k, t) PRISM EvoX(32, 60)(4, 196) OpenEvolve (32, 171)(36, 22) AdaEvolve (40, 136)(7, 115) Cloudcast EvoX(40, 134)(6, 134) OpenEvolve (40, 158)(5, 162) AdaEvolve(40, 83)(7, 114) Txn EvoX(40, 158)(4, 198) OpenEvolve (40, 159)(13, 60) AdaEvolve(38, 40)(19, 40) Table 1: Optimal budget split(k, t)withkt≤ Bat the full and≈10% budgets. Allocations vary by strategy and budget. On PRISM at 10%, prescriptions are op- posite (EvoX deep/narrow, OpenEvolve wide/shallow). AdaEvolve on transaction scheduling peaks at40itera- tions; further depth is wasted. The ranking inverts with the budget. Table 2 gives each strategy at one seed and at forty seeds, both at full depth. On Cloudcast the order reverses: EvoX is last at one seed and first at forty seeds, and OpenEvolve moves from second to last. On trans- action scheduling EvoX and OpenEvolve swap the second and third places. On PRISM the single- seed scores are within0.03(a near tie) while at forty seeds AdaEvolve leads by0.53. These cross- ings arise because strategies use budgets differently: added seeds help AdaEvolve far more than EvoX on PRISM, while EvoX gains on both axes on Cloudcast, so single-point measurements observe only one corner of this surface. Figure 3 turns this into a probability: small budgets yield unreliable rankings; while more depth corrects the order late, more seeds fix it steadily. On every task a large part of the budget is needed before the order is TaskStrategy1 seed40 seeds PRISM EvoX26.254 (2) 26.256 (2) OpenEvolve 26.240 (3) 26.256 (2) AdaEvolve 26.285 (1) 26.788 (1) Cloudcast (×10 −3 ) EvoX1.19 (3) 1.82 (1) OpenEvolve 1.33 (2)1.52 (3) AdaEvolve 1.50 (1)1.59 (2) Txn EvoX4255 (2)4386 (3) OpenEvolve 4229 (3)4973 (2) AdaEvolve 4766 (1) 21277 (1) Table 2: Expected bestcombined_scoreat one and forty seeds (full depth), with rank in parentheses (higher is better). The order changes with the seed count on every task. reliable. More seeds also surfaced an AdaEvolve program scoring over4×any prior result, a valida- tor exploit, not a better scheduler (Appendix D); that more seeds expose such flaws is part of our argument. 5 Conclusion Evaluating LLM evolutionary search at one budget point is not enough. The best split between seeds and iterations, and even which method wins, de- pend on the strategy, the task, and the total budget. We recommend three practices. Report the seeds- by-iterations frontier and the optimal split at each budget, not a single score. Run several seeds, since single-seed rankings are unreliable. Tune depth per strategy and task rather than fixing iterations by habit. All three follow from replaying logged trajectories, so they cost no extra runs. 48 1632486496 128160200 iterations budget t (depth, all seeds) 0.0 0.2 0.4 0.6 0.8 1.0 P(declared order ≠ full-budget order) 124816243240 seed budget k (width, 200 iters) 0.0 0.2 0.4 0.6 0.8 1.0 200400800 16003200480064008000 total budget B = k×t evals (best split) 0.0 0.2 0.4 0.6 0.8 1.0 When the budget flips the ranking PRISMCloudcastTxnCircle packingHeilbronn Figure 3: Probability that the strategy order declared at a partial budget differs from the full-budget order, one curve per task. Left: iterations at full width. Middle: seeds at full depth. Right: total budget at its best split. Every curve falls to zero as the budget approaches full. The order settles late with depth but steadily with more seeds. Shaded bands are bootstrap 95% confidence intervals. 4 Limitations While this study introduces a rigorous assessment methodology necessary for validating improve- ments in evolutionary processes, we acknowledge that this protocol imposes a higher computational and financial barrier. Evaluating a full grid of seeds and iterations requires significantly more LLM API calls per assessment than traditional single-point evaluations. Additionally, our budget-grid protocol is an offline evaluation tool designed to accurately rank methods post-hoc. While it reveals that the optimal width-versus-depth split is task-dependent, a prediction of this optimal split, without first ex- ploring the grid, is still an open question. References Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Ar- nav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. 2026. Gepa: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representa- tions, volume 2026, pages 8479–8565. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirho- seini. 2024. Large language monkeys: Scaling infer- ence compute with repeated sampling. arXiv preprint arXiv:2407.21787. Mert Cemri, Shubham Agrawal, Akshat Gupta, Shu Liu, Audrey Cheng, Qiuyang Mang, Ashwin Naren, Lutfi Eren Erdogan, Koushik Sen, Matei Za- haria, et al. 2026.Adaevolve: Adaptive llm driven zeroth-order optimization. arXiv preprint arXiv:2602.20133. Audrey Cheng, Lynn Liu, Shu Wang, Ion Stoica, et al. 2025. Barbarians at the gate: How ai is upending systems research. arXiv preprint arXiv:2510.06189. Marquita Ellis and Paul Castro. 2026. Don’t gamble, gamble: An analytical framework for ai-driven re- search systems. arXiv preprint arXiv:2606.02863. Chrisantha Fernando,Dylan Banarse,Henryk Michalewski, Simon Osindero, and Tim Rock- täschel. 2023.Promptbreeder: Self-referential self-improvement via prompt evolution.arXiv preprint arXiv:2309.16797. Yonatan Gideoni, Sebastian Risi, and Yarin Gal. 2026. Simple baselines are competitive with code evolution. arXiv preprint arXiv:2602.16805. Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2024. Connecting large language models with evolutionary algorithms yields powerful prompt op- timizers. In International Conference on Learning Representations (ICLR). Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 32. Yuichi Inoue, Kou Misaki, Yuki Imajuku, So Kuroki, Taishi Nakamura, and Takuya Akiba. 2026. Wider or deeper? scaling llm inference-time compute with adaptive branching tree search. Advances in Neural Information Processing Systems, 38:35448–35484. Yoichi Ishibashi, Taro Yano, and Masafumi Oya- mada. 2026. Effective harness engineering for algo- rithm discovery with coding agents. arXiv preprint arXiv:2605.15221. Joshua Kazdan, Rylan Schaeffer, Youssef Allouah, Colin Sullivan, Kyssen Yu, Noam Levi, and Sanmi Koyejo. 2025.Efficient prediction of pass@ k scaling in large language models. arXiv preprint arXiv:2510.05197. Robert Lange, Yuki Imajuku, and Edoardo Cetin. 2026. Shinkaevolve: Towards open-ended and sample- efficient program evolution. In International Con- ference on Learning Representations, volume 2026, pages 74026–74078. Fei Liu, Tong Xialiang, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. 2024. Evolution of heuristics: Towards efficient auto- matic algorithm design using large language models. In International Conference on Machine Learning (ICML). ArXiv:2401.02051. Shu Liu, Shubham Agarwal, Monishwaran Maheswaran, Mert Cemri, Zhifei Li, Qiuyang Mang, Ashwin Naren, Ethan Boneh, Audrey Cheng, Melissa Z Pan, et al. 2026a. Evox: Meta-evolution for automated discovery. arXiv preprint arXiv:2602.23413. Shu Liu, Mert Cemri, Shubham Agarwal, Alexander Krentsel, Ashwin Naren, Qiuyang Mang, Zhifei Li, Akshat Gupta, Monishwaran Maheswaran, Audrey Cheng, et al. 2026b. Skydiscover: A flexible, adap- tive framework for ai-driven scientific and algorith- mic discovery. In Proceedings of the ACM Confer- ence on AI and Agentic Systems, pages 1223–1227. Yecheng Jason Ma, William Liang, Guanzhi Wang, De- An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2024. Eu- reka: Human-level reward design via coding large language models. In International Conference on Learning Representations (ICLR). Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in neural information pro- cessing systems, 36:46534–46594. 5 Alexander Novikov, Ngân V ̃ u, Marvin Eisenberger, Em- ilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abi- gail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. 2025. Alphaevolve: A coding agent for scientific and algorithmic discovery. Technical report, Google DeepMind. BernardinoRomera-Paredes,Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. 2024. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475. Rylan Schaeffer, Joshua Kazdan, John Hughes, Jor- dan Juravsky, Sara Price, Aengus Lynch, Erik Jones, Robert Kirk, Azalia Mirhoseini, and Sanmi Koyejo. 2025. How do large language monkeys get their power (laws)? arXiv preprint arXiv:2502.17578. Asankhaya Sharma. 2025.Openevolve: An open- source implementation of alphaevolve.https:// github.com/codelion/openevolve. James E Smith and Robert L Winkler. 2006. The op- timizer’s curse: Skepticism and postdecision sur- prise in decision analysis. Management Science, 52(3):311–322. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Ku- mar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. Preprint, arXiv:2408.03314. Haoran Ye, Jiarui Wang, Zhiguang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guo- jie Song. 2024. Reevo: Large language models as hyper-heuristics with reflective evolution. Advances in neural information processing systems, 37:43571– 43608. A Implementation details All experiments run on SkyDiscover (Liu et al., 2026b), an open framework for AI-driven algorith- mic discovery, using its strategy implementations, task evaluators, and logging. Building on it keeps our pipeline reproducible. B Budget frontier Figure 4 plots the best expected score against the total budget, using the optimal split at each bud- get. The gaps between strategies open and close as the budget grows, and the leader changes on some tasks, the effect Figure 3 quantifies. Figure 4 keeps every seed, so a few lucky seeds set the top of the best-of-k and can mask the typical trend. Figure 5 repeats it with each technique’s best seed dropped (top two on transaction scheduling, top one on the other tasks, and the best draw of the no-evolve baseline). Read it for the trend, not for absolute scores. 6 020406080100 seed budget used (%) 26.3 26.4 26.5 26.6 26.7 26.8 mean best combined_score (best-of-k over seeds) PRISM (serving goodput) EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) 020406080100 seed budget used (%) 0.0012 0.0013 0.0014 0.0015 0.0016 0.0017 0.0018 Cloudcast (broadcast cost) EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) 020406080100 seed budget used (%) 5000 7500 10000 12500 15000 17500 20000 Txn scheduling (throughput) EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) 020406080100 iterations budget used (%) [t=1 ramp clipped] 25.8 25.9 26.0 26.1 26.2 26.3 mean best combined_score (best-so-far over iterations) EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) 020406080100 iterations budget used (%) [t=1 ramp clipped] 0.0008 0.0009 0.0010 0.0011 0.0012 0.0013 0.0014 0.0015 EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) 020406080100 iterations budget used (%) [t=1 ramp clipped] 3800 4000 4200 4400 4600 4800 EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) 020406080100 total budget used (%) [k×t evals, best split] 23 24 25 26 mean best combined_score (best split of k seeds × t iters) EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) no-evolve best-of-N (valid) (N=7827 valid / 8000 calls) 020406080100 total budget used (%) [k×t evals, best split] 0.0008 0.0010 0.0012 0.0014 0.0016 0.0018 EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) no-evolve best-of-N (valid) (N=4898 valid / 8000 calls) 020406080100 total budget used (%) [k×t evals, best split] 5000 7500 10000 12500 15000 17500 20000 EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) no-evolve best-of-N (N=2592 draws / 8000 calls) Where the budget pays off Figure 4: Best expectedcombined_scoreagainst the LLM-call budget, one column per task. Top: value of more seeds at full depth. Middle: value of more iterations for one seed. Bottom: the frontier, the best score at each total budget using the optimal split. The gaps between strategies open and close as the budget grows, so the strategy to prefer depends on how many calls are available. The no-evolve best-of-N line is a reference, not part of the strategy comparison. Shaded bands span the 25th–75th percentile of the best-of-k distribution. 7 020406080100 seed budget used (%) 26.25 26.30 26.35 26.40 26.45 26.50 26.55 26.60 mean best combined_score (best-of-k over seeds) PRISM (serving goodput) EvoX (n=39) OpenEvolve (n=39) AdaEvolve (n=39) 020406080100 seed budget used (%) 0.0012 0.0013 0.0014 0.0015 0.0016 Cloudcast (broadcast cost) EvoX (n=39) OpenEvolve (n=39) AdaEvolve (n=39) 020406080100 seed budget used (%) 4150 4200 4250 4300 4350 Txn scheduling (throughput) EvoX (n=38) OpenEvolve (n=38) AdaEvolve (n=38) 020406080100 iterations budget used (%) [t=1 ramp clipped] 25.8 25.9 26.0 26.1 26.2 mean best combined_score (best-so-far over iterations) EvoX (n=39) OpenEvolve (n=39) AdaEvolve (n=39) 020406080100 iterations budget used (%) [t=1 ramp clipped] 0.0007 0.0008 0.0009 0.0010 0.0011 0.0012 0.0013 0.0014 0.0015 EvoX (n=39) OpenEvolve (n=39) AdaEvolve (n=39) 020406080100 iterations budget used (%) [t=1 ramp clipped] 3800 3900 4000 4100 4200 4300 EvoX (n=38) OpenEvolve (n=38) AdaEvolve (n=38) 020406080100 total budget used (%) [k×t evals, best split] 22.5 23.0 23.5 24.0 24.5 25.0 25.5 26.0 26.5 mean best combined_score (best split of k seeds × t iters) EvoX (n=39) OpenEvolve (n=39) AdaEvolve (n=39) no-evolve best-of-N (valid) (N=7826 valid / 8000 calls) 020406080100 total budget used (%) [k×t evals, best split] 0.0008 0.0010 0.0012 0.0014 0.0016 EvoX (n=39) OpenEvolve (n=39) AdaEvolve (n=39) no-evolve best-of-N (valid) (N=4897 valid / 8000 calls) 020406080100 total budget used (%) [k×t evals, best split] 3400 3600 3800 4000 4200 EvoX (n=38) OpenEvolve (n=38) AdaEvolve (n=38) no-evolve best-of-N (N=2590 draws / 8000 calls) Where the budget pays off (reference) Figure 5: Same as Figure 4 with the best seed(s) per technique removed. Trend-reading only. 8 TaskStrategy1 seed40 seeds Circle Packing EvoX 2.6141 (1) 2.6360 (1) OpenEvolve 2.5472 (3) 2.6234 (3) AdaEvolve 2.6137 (2) 2.6358 (2) Heilbronn (×10 −2 ) EvoX2.955 (2) 3.523 (1) OpenEvolve 2.931 (3)3.388 (3) AdaEvolve 3.039 (1)3.498 (2) Table 3: Expected best score at one and forty seeds (full depth), rank in parentheses (higher is better). On Heilbronn, AdaEvolve leads at one seed but EvoX at forty; circle packing is a near tie. C Math tasks The two math tasks show the same budget effects as the systems tasks (Figure 3). Figure 6 plots the seeds-by-iterations surface: the best cell again lands in different places across strategies, and depth saturates before the full budget. Figure 7 shows the frontier rising with the budget, with the gaps between strategies opening and closing as before. Table 3 confirms the ranking inversion: on Heil- bronn, AdaEvolve leads at one seed but EvoX over- takes it at forty, exactly the crossing a single-seed protocol would miss; circle packing demonstrates a near tie between EvoX and AdaEvolve at both budgets. Both tasks support the same conclusion as the systems tasks. D The discovered transaction scheduler The transaction scheduling column of Table 2 looks like our strongest result, but part of it is an artifact of the evaluator. The validity check, validate_schedule, only tests that the returned sequence is a permutation of the indices it con- tains. It never checks that every transaction was scheduled. Each workload holds100transactions, so a program can drop some of them, lower its makespan, and still pass with validity= 1.0. Running more seeds exposed this gap, and dif- ferent strategies use it to different degrees (Ta- ble 4). AdaEvolve is the extreme case: its best pro- gram scorescombined_score 21277(makespan 46), about4×any prior result, by scheduling a sin- gle transaction per workload and skipping the other 99. Two of its forty seeds do this. OpenEvolve’s best program (4973) is subtler: it drops about45of the100transactions in one workload and completes the other two, which is why its makespan (200) sits below EvoX’s (227). Only EvoX schedules every transaction. We score under the official evaluator to stay com- parable with prior work, so we keep every program in the tables and figures. But it means the leader- board on this task partly reflects how much of the workload a program quietly drops, not how well it schedules. A sound evaluator would need a cover- age check, which this benchmark lacks. We think this makes our point stronger. A one- or three-seed run would rarely reach these programs, while forty seeds surface them, change the ranking, and stress-test the benchmark itself, exposing an evaluator flaw a small budget would miss. 9 125102040 10 20 50 100 200 EvoX iterations t (log) Circle packing · EvoX 80 200 500 1000 2000 4000 125102040 10 20 50 100 200 Heilbronn · EvoX 80 200 500 1000 2000 4000 125102040 10 20 50 100 200 OpenEvolve iterations t (log) Circle packing · OpenEvolve 80 200 500 1000 2000 4000 125102040 10 20 50 100 200 Heilbronn · OpenEvolve 80 200 500 1000 2000 4000 125102040 seeds k (log) 10 20 50 100 200 AdaEvolve iterations t (log) Circle packing · AdaEvolve 80 200 500 1000 2000 4000 125102040 seeds k (log) 10 20 50 100 200 Heilbronn · AdaEvolve 80 200 500 1000 2000 4000 2.57 2.58 2.59 2.60 2.61 2.62 2.63 mean best sum of radii 2.50 2.52 2.54 2.56 2.58 2.60 mean best sum of radii 2.624 2.626 2.628 2.630 2.632 2.634 mean best sum of radii 0.028 0.030 0.032 0.034 mean best min triangle area (norm.) 0.028 0.029 0.030 0.031 mean best min triangle area (norm.) 0.031 0.032 0.033 0.034 mean best min triangle area (norm.) Best score over the seeds x iterations grid Figure 6: Seeds-by-iterations surface for the two math tasks; the best cell is marked. As in the systems tasks, it moves across strategies and depth saturates early. 10 020406080100 seed budget used (%) 2.54 2.56 2.58 2.60 2.62 2.64 mean best raw task score (best-of-k over seeds) Circle packing (sum of radii) EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) 020406080100 seed budget used (%) 0.028 0.029 0.030 0.031 0.032 0.033 0.034 0.035 Heilbronn (min triangle area) EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) 020406080100 iterations budget used (%) [t=1 ramp clipped] 2.2 2.3 2.4 2.5 2.6 mean best raw task score (best-so-far over iterations) EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) 020406080100 iterations budget used (%) [t=1 ramp clipped] 0.010 0.015 0.020 0.025 0.030 EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) 020406080100 total budget used (%) [k×t evals, best split] 1.2 1.4 1.6 1.8 2.0 2.2 2.4 2.6 mean best raw task score (best split of k seeds × t iters) EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) no-evolve best-of-N (N=7530 draws / 8000 calls) 020406080100 total budget used (%) [k×t evals, best split] 0.000 0.005 0.010 0.015 0.020 0.025 0.030 0.035 EvoX (n=40) OpenEvolve (n=40) AdaEvolve (n=40) no-evolve best-of-N (N=6948 draws / 8000 calls) Where the budget pays off Figure 7: Best expected score against the budget for the math tasks. Top: seeds at full depth. Middle: iterations for one seed. Bottom: the frontier at the optimal split. 11 StrategyBest combined_scoreMakespanTransactions scheduledFull workload? EvoX4386227300/300yes OpenEvolve4973≈200≈255/300no (drops≈45) AdaEvolve21277463/300no (drops 297) Table 4: The best transaction-scheduling program found by each strategy at forty seeds, under the official evaluator. Each of the three workloads holds100transactions, for300in total. AdaEvolve and OpenEvolve both pass the validity check while leaving transactions unscheduled, which lowers their makespan and inflatescombined_score; only EvoX schedules the full workload. The evaluator accepts all three becausevalidate_schedulenever checks coverage. 12