Paper deep dive

Meta-Reflective Evolution Effectively Improves Production AI Systems

Seung Won Wilson Yoo, Chang Zeng, Aditi Somayajula, Sahithi Ankireddy, Rohan Kosalge, Avishree Khare, John Mitchell, Aditya G. Parameswaran, Rajhans Samdani, Anupam Datta

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 287

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 10/12/2026, 6:21:32 AM

Summary

The paper introduces CoCoEvolve (Cortex Code Evolve), an evolutionary optimizer for production AI systems that utilizes 'meta-reflection'. Unlike traditional methods that mutate based only on the immediate parent's failure, CoCoEvolve's agentic mutator (Cortex Code) retrieves evidence from the entire evolutionary history and probes the live system to verify and propose mutations. It achieves state-of-the-art results on benchmarks like DABStep (92.8% accuracy), data-eng-bench, and HotpotQA, outperforming baselines such as Meta-Harness and GEPA.

Entities (9)

CoCoEvolve · method · 98%Cortex Code · softwaretool · 95%DABStep · benchmark · 95%Meta-Reflective Evolution · concept · 95%GEPA · baseline-method · 92%Meta-Harness · baseline-method · 92%data-eng-bench · benchmark · 90%HotPotQA · benchmark · 90%MAP-Elites · algorithmpolicy · 85%

Relation Signals (6)

CoCoEvolve → achieveshighaccuracyon → DABStep

confidence 95% · On DABStep, CoCoEvolve tops the public leaderboard, reaching 92.8% held-out hard accuracy

CoCoEvolve → uses → Cortex Code

confidence 95% · CoCoEvolve uses Cortex Code as its coding-agent mutator

CoCoEvolve → usesmechanism → Meta-Reflective Evolution

confidence 95% · We introduce CoCoEvolve ... an evolutionary optimizer built around what we call meta-reflection

CoCoEvolve → achieveshighsolverateon → data-eng-bench

confidence 90% · On data-eng-bench, CoCoEvolve lifts test-set solve rate from 59.1% to 77.4%

CoCoEvolve → outperforms → Meta-Harness

confidence 90% · On DABStep, CoCoEvolve tops the public leaderboard, reaching 92.8% held-out hard accuracy compared to 78.4% for Meta-Harness

CoCoEvolve → outperforms → GEPA

confidence 90% · 30.4% for GEPA under matched conditions

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large Language Model (LLM)-guided evolutionary optimization has successfully improved prompts, programs, and even entire AI systems. Most optimization methods select a candidate from the search frontier, then mutate it in one LLM pass based only on its failures. This focus on the frontier can stall on production AI systems, where isolated failures reveal little about complex, hidden interactions. We introduce CoCoEvolve (Cortex Code Evolve), an evolutionary optimizer built around what we call meta-reflection, wherein an agentic mutator (Cortex Code) selectively retrieves evidence from the entire evolutionary history and probes the live system before verifying and proposing mutations. We apply CoCoEvolve to optimize three classes of production AI systems: data agents, LLM-powered data pipelines, and data engineering workflows. On DABStep, CoCoEvolve tops the public leaderboard, reaching 92.8% held-out hard accuracy compared to 78.4% for Meta-Harness and 30.4% for GEPA under matched conditions. Across six LLM data pipeline workloads, it outperforms every evaluated baseline. On data-eng-bench, CoCoEvolve lifts test-set solve rate from 59.1% to 77.4%, past the best baseline of 73.6%. It also has 11 percentage points higher accuracy than GEPA on HotpotQA while remaining competitive with specialized optimizers on established mathematical and systems benchmarks. Matched ablations confirm that each component of CoCoEvolve contributes to these gains, demonstrating the value of meta-reflection for optimizing production AI systems.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

283,405 characters extracted from source content.

Expand or collapse full text

META-REFLECTIVE EVOLUTION EFFECTIVELY IMPROVES PRODUCTION AI SYSTEMS Seung Won Wilson Yoo 1 , Chang Zeng 2,∗ , Aditi Somayajula 3,∗ , Sahithi Ankireddy 3,∗ Rohan Kosalge 4,∗ , Avishree Khare 5,∗ , John Mitchell 1,3 , Aditya G. Parameswaran 1,4 Rajhans Samdani 1 , Anupam Datta 1 1 Snowflake 2 UMass Amherst 3 Stanford 4 UC Berkeley 5 UPenn ABSTRACT Large Language Model (LLM)-guided evolutionary optimization has successfully improved prompts, programs, and even entire AI systems. Most optimization methods select a candidate from the search frontier, then mutate it in one LLM pass based only on its failures. This focus on the frontier can stall on produc- tion AI systems, where isolated failures reveal little about complex, hidden in- teractions. We introduce CoCoEvolve (Cortex Code Evolve), an evolutionary optimizer built around what we call meta-reflection, wherein an agentic mutator (Cortex Code) selectively retrieves evidence from the entire evolutionary history and probes the live system before verifying and proposing mutations. We ap- ply CoCoEvolve to optimize three classes of production AI systems: data agents, LLM-powered data pipelines, and data engineering workflows. On DABStep, CoCoEvolve tops the public leaderboard, reaching 92.8% held-out hard accuracy compared to 78.4% for Meta-Harness and 30.4% for GEPA under matched con- ditions. Across six LLM data pipeline workloads, it outperforms every evaluated baseline. On data-eng-bench, CoCoEvolve lifts test-set solve rate from 59.1% to 77.4%, past the best baseline of 73.6%. It also has 11p higher accuracy than GEPA on HotpotQA while remaining competitive with specialized optimizers on established mathematical and systems benchmarks. Matched ablations confirm that each component of CoCoEvolve contributes to these gains, demonstrating the value of meta-reflection for optimizing production AI systems. 1INTRODUCTION Large language models (LLMs) are increasingly used as semantic mutation operators: given a candi- date prompt or program and evaluation feedback for that candidate, an LLM proposes a revised can- didate. This approach, exemplified by evolutionary optimizers like AlphaEvolve and OpenEvolve, has led to breakthrough improvements in prompts, algorithms, and other text-based artifacts be- cause LLMs can make semantically meaningful changes that conventional random mutation cannot (Novikov et al., 2025; Sharma, 2025). However, most LLM-guided evolutionary optimizers select a single parent, use its failures to propose a new candidate in one LLM call, and only then evaluate the result. This parent-local, one-shot interface is effective when the failure makes both the defect and its repair apparent. It falls short for production AI systems, where isolated failures can reveal little about interacting causes and relevant evidence may be distributed across the search history. First, the selected parent’s failure may not reveal the root cause, and a locally beneficial edit may regress capabilities elsewhere in the search history. Second, the mutator cannot interact with the target environment, test hypotheses, or revise its candidate in response to intermediate results. Search can consequently trade one solved task for another, revisit exhausted strategies, or end up plateauing (Zhou et al., 2026). We introduce CoCoEvolve (Cortex Code Evolve), an evolutionary optimizer built around meta- reflection: (i) selectively retrieving evidence from the accumulated search history beyond the se- ∗ Work done during internship at Snowflake. 1 arXiv:2610.12794v1 [cs.LG] 8 Oct 2026 Parent-local mutation Mutation starts with A Parent 28 / 300 + × Parent’s feedback “Belles total is wrong.” å Builder · one-shot LLM mutator Two possible explanations wrong set of feeswrong fee-combination rule j LLM picks: try a new set of fees v Single LLM completion edits agent specification No probe distinguishes the alternatives ; no history query · no live probe q Commit child→ verify 28→ 28 / 300 correct (no change) ² Meta-reflective mutation Mutation starts with A Parent 28 / 300 + [ Strategy Ledger + × Parent’s feedback “Belles total is wrong.” å Builder · agentic mutator Î Evolutionary history Tools to query history Select relative (previously evaluated program) ↓ retrieve trace Prior trace suggests: • all matching fees stack Û Coding-agent investigation Why is the total wrong? 1 inspect parent + relative 2 compare hypotheses 3 implement recovered rule õ Live environment Live SQL queries on warehouse data Read Documentation Refine which fees match Replay Belles transactions Try highest matching fee — Incorrect Try stacking all matching fees — Correct U Recovered rule stack all matching fees ¥ Smoke test→ agent runs on sample inputs✓ ¥ Commit child→ verify 28→ 95 / 300 correct (+67) Figure 1: Left: given under-determined feedback, the parent-local mutator guesses a repair in one call and makes no progress. Right: a meta-reflective mutator queries the evolutionary history and reads the trace of a previously evaluated program to determine that all matching fees stack. It also reads documentation to refine which fees match, and synthesizes the two observations. It tests the resulting hypothesis before commit and raises training accuracy from 28/300 to 95/300. lected parent’s failure and (i) actively probing the target environment while constructing and val- idating a mutation. To support meta-reflection, CoCoEvolve uses Cortex Code as its coding-agent mutator (Snowflake Inc., 2026b). The mutator can inspect the parent, run focused experiments, revise its implementation, and validate the result before committing a child. It accesses evolution- ary history through both a compact summary and structured queries, allowing it to retrieve prior programs that solved a specific task, inspect the mutation that restored a lost capability, or identify related tasks that have historically regressed together. Figure 1 illustrates meta-reflection for a DABStep data agent (Egg et al., 2025), where payment- processing rules must be inferred across documentation and live data. Here, a one-shot, parent- local mutation doesn’t make progress, whereas CoCoEvolve’s meta-reflective mutation retrieves the missing evidence, validates the corresponding repair, and succeeds. The same behavior repeats with Lockpicking with Interference, a synthetic stress test where every leading baseline plateaus below perfect accuracy within 50 iterations, while CoCoEvolve reaches 100% in 6 iterations (Ap- pendix G.1). We evaluate CoCoEvolve primarily on production AI systems, where correctness depends on inter- actions with live systems rather than prompt text alone. These include agent specifications coupled to tools and semantic data models, SQL pipelines that invoke LLMs to process textual data (Liskowski et al., 2025), and data-engineering workflows (dbt Labs, 2026). We additionally evaluate standard prompt, mathematical, and systems optimization benchmarks from prior work to test whether the method remains competitive when interaction with the environment is limited or unnecessary. CoCoEvolve optimizes an agent specification on DABStep (Egg et al., 2025), reaching 92.8% held- out hard accuracy, versus 78.4% for an interactive Meta-Harness baseline and 30.4% for GEPA (Lee et al., 2026; Agrawal et al., 2026b); it also tops the validated public leaderboard. 1 CoCoEvolve also improves over the strongest evaluated baseline on six LLM-powered data processing pipeline workloads by up to 10 p. On held-out HotpotQA, it gains 11 p over GEPA and 43 p over the seed (Yang et al., 2018; Agrawal et al., 2026b). On data engineering, it achieves the strongest 1 At submission, this score has not been externally validated; we have contacted the DABStep maintainers but have not received a response. 2 Initialize Repeat Until Budget Exhausted Archive A A A Parent Selection ⋆ Score Ø Observation Î History Z Curator (Context Curation) Context Bundle to prompt Environment Tools to mutator Builder (Agentic Mutator) « Skill Use ¥ Self-Test Æ Agentic Mutator õ Structural Edit õ Û Query History Live Environment / Target Systems Å á A Child Program Evaluator À Better Than Niche Occupant? Discard Admit to Archive Figure 2: CoCoEvolve proposes a new candidate in each iteration by selecting a parent from the archive, curating evidence from the search history to include in the prompt, and equipping a coding- agent mutator with on-demand tools for querying prior lineages and the live target environment. A coding agent investigates, edits, and self-tests a child; the evaluator scores it, records the resulting observations in the history, and, under the default MAP-Elites policy, either discards it or admits it to the archive if it improves its niche. This loop combines compact prompt-visible evidence with environment feedback while preserving diverse programs for future selection. overall performance among the evaluated evolutionary methods and leads on held-out data-eng- bench (Snowflake AI Research, 2026), solving 77.4% of tasks. Across mathematical and systems optimization objectives, it is competitive with leading evolutionary methods such as AdaEvolve (Cemri et al., 2025), with the largest improvements occurring on objectives that exhibit extended search plateaus. Matched ablations confirm that each component of CoCoEvolve contributes to these gains. 2COCOEVOLVE We introduce CoCoEvolve, an evolutionary optimizer built around meta-reflection (Section 2.2). Figure 2 summarizes its evolutionary loop and Algorithm 1 in Appendix C specifies it in full. Our implementation instantiates the coding-agent mutator with Cortex Code. 2.1PROBLEM FORMULATION We treat the object of optimization as a program p — code, a prompt, a configuration, an agent specification, or any other text-representable artifact — drawn from a program space P . Let T denote the task space andD a distribution over it. An evaluator f :P ×T → R×O, f (p,τ ) = s(p,τ ), o(p,τ ) returns both a scalar score s(p,τ ) ∈ R and an observation o(p,τ ) ∈ O: the full record produced while computing that score, including tool calls, intermediate outputs, executed queries, and error messages. The optimization goal is to find p ∗ = arg max p∈P Φ(p) within T iterations, where Φ(p) = E τ∼D [s(p,τ )] represents the aggregate fitness of a program. CoCoEvolve pursues this goal via an iterative evolutionary loop. After iteration t−1, it maintains an archiveA t−1 of previously evaluated candidate programs. At iteration t, the harness selects a parent program p ′ t ∈ A t−1 and produces a child p t . Most prior methods condition mutation only on the selected parent p ′ t and its observations; CoCoEvolve instead uses the full search history accumulated so far, denoted by H t−1 =(p i ,T i ,S i ,O i ) i<t , Here, T i ⊆ T is the subset selected to evaluate p i , with S i := s(p i ,τ ) | τ ∈ T i and O i := o(p i ,τ ) | τ ∈ T i denoting the resulting sets of scores and observations. This history sup- ports the mutation policy developed next: context curation exposes selected evidence from H t−1 , 3 environment augmentation provides tools for retrieving additional evidence on demand, and the coding-agent mutator uses both to construct the child. 2.2META-REFLECTION We define meta-reflection as reasoning beyond one parent’s observations by selectively retrieving historical evidence from the accumulated search history and actively acquiring feedback from the target environment. Formally, a meta-reflective mutator receives two complementary inputs. First, a compact context bundle c t , curated from the search history, is injected directly into the mutator’s prompt. Second, an environmentE t provides tools that the mutator invokes on demand. It augments a base environment E 0 ⊆E of execution, validation, and target-system tools with selective access to the search history. c t = Curate(H t−1 )∈C, E t = Augment(E 0 ;p ′ t ,H t−1 )⊆E. A coding-agent mutator then performs meta-reflection by reasoning over c t and gathering additional evidence fromE t as needed while constructing and validating the child p t before committing it. p t ∼ MetaReflect(p ′ t ,c t ;E t ). Relation to prior reflective mutation. This interface recovers prior reflective mutation as the spe- cial case in which c t = O ′ t ,and E t = E 0 = ∅: each revision conditions only on that trace or side information (Agrawal et al., 2026a). Additionally, these reflective methods implement mutation with a single model call over this preassembled evidence. In CoCoEvolve, these interface components take the following concrete forms. Ledger to gauge stagnation. CoCoEvolve’s Curate operator constructs a compact ledger of recent strategies and their outcomes (Appendix C.1). The agent uses this ledger to assess stagnation and choose whether to continue the current strategy, adopt a newly proposed one, or revert to undirected mutation. AdaEvolve makes a related search-level decision through a fixed trigger, invoking tactic generation when improvement falls below a threshold (Cemri et al., 2025). CoCoEvolve instead lets the same class of model that performs mutation reason over concurrently tracked strategies and make this decision from the search trajectory, without a hand-tuned stagnation trigger. Live environment interaction. The base environmentE 0 provides domain-specific tools for execu- tion, testing, and target-system queries. For example, in mathematical and systems optimization,E 0 exposes scratch runners that reveal syntax and execution errors. In DABStep agent optimization, it provides tools for SQL execution, scratch-agent deployment, target-question tests, and deployment checks. This feedback lets the agent diagnose and repair failures before committing a child to outer- loop evaluation. Appendix C.2 specifies the interfaces, and Appendix D.2 compares static, edit-only agentic, and self-validating mutation. Tools to query interlinked history. Beyond these domain capabilities, Augment forms E t by adding on-demand tools that selectively query the interlinked program archive as evidence is needed. These tools expose per-task lineage: which program first solved a task, whether a later mutation lost that capability, and which tasks tend to co-regress. The agent can therefore retrieve the exact diff that last restored a lost capability or vet a candidate child against known co-regressions before committing. Appendix C.2 and Table 6 specify the complete query interface, while Appendix D.3 isolates the contributions of the injected ledger and history-query tools. After validation, we return the generated child p t to the outer search loop, where f scores it, H t records it, and the search policy governs archive admission and subsequent parent selection. 2.3SEARCH POLICY By default, CoCoEvolve uses standard MAP-Elites parent selection (Sharma, 2025; Mouret & Clune, 2015). For objectives scored over a pool of questions or test cases, we introduce a novel bud- geted ℓ p roster that prioritizes complementary coverage; Appendix C.5 defines it, and Appendix D.1 compares it with MAP-Elites, best-only selection, and pareto-frontier sampling. Beyond parent selection, an optional similarity-informed task-selection policy uses proxy scores to reduce evaluation cost and focus mutation on high-information tasks. 4 Similarity-guided sparse evaluation and observation curation. Prior methods typically expose the mutator to observations from a random task minibatch. When the task pool V is large or individ- ual evaluations are costly, however, exhaustively scoring every candidate is impractical. On DAB- Step, for example, each agent query takes roughly 3–10 minutes, so scoring the full 300-question search pool requires 15–50 agent-hours before parallelism. At the same time, tasks need not be independent: a repair that equips a candidate to solve one task may transfer to other tasks requiring the same capability. This motivates sparse, targeted evaluation: score a bounded subset, infer likely performance on the remaining tasks, and direct the next mutation toward an unresolved target with broad expected transfer. CoCoEvolve realizes this idea with a semantic task graph. Let κ : V × V → [0, 1] measure task similarity. Using only evaluated tasks, it forms a Nadaraya–Watson kernel estimate of the candidate’s score on τ (Nadaraya, 1964; Watson, 1964): bs(p,τ ) = P τ ′ ∈tested(p) κ(τ,τ ′ )s(p,τ ′ ) P τ ′ ∈tested(p) κ(τ,τ ′ ) , so evaluated neighbors provide a surrogate expected score for tasks not yet tested. These surrogate scores also determine where mutation effort is likely to be most informative. Each task not solved by the parent receives an information-gain score: gain(τ ) = 1−bs(p,τ ) δ(τ ) X τ ′ ∈V κ(τ,τ ′ ). Here, δ(τ ) downweights repeatedly unsuccessful targets; sampling by gain then focuses c t on likely failures with broad semantic reach. This policy is a task-level counterpart to AdaEvolve’s island- level Dynamic Exploration Intensity (Cemri et al., 2025): it directs scarce evaluation and mutation effort toward failures with the greatest estimated transfer. Appendix C.3 details the policy, Table 7 records where it is active, and matched DABStep ablations showcase its value (Appendix D.4). 3EXPERIMENTS Where applicable, we follow a standard train/validation/test split. CoCoEvolve has full access to training examples during optimization. Validation scores may be used to guide search, but the con- tent of validation instances is never exposed to the optimizer directly. Final results are reported on the held-out test split. We evaluate CoCoEvolve on four task categories: agent optimization, AI Functions, data engineering, and mathematical/system/prompt optimization. Parent-selection vari- ants of the archive rule are specified in Appendix C and compared in Appendix D.1; the remaining mechanism-ablation protocol is Appendix D.2–D.4. Evaluator details for the math and systems suite appear in Appendix I. 3.1AGENT OPTIMIZATION Major enterprise platforms increasingly deploy fully managed agentic frameworks inside the data perimeter (Snowflake Inc., 2026a; Databricks, 2024; Amazon Web Services, 2023). These agents iteratively decompose requests and use intermediate results to orchestrate semantic data access, unstructured retrieval, sandboxed code execution, and custom APIs. Although the platform hides infrastructure and state management, reliability depends on the instructions, tool interfaces, and data models that guide this reasoning, making each a target for optimization. Setup. We formulate agent optimization as searching over agent specifications p∈P encompassing system prompts, tool schemas, and planning instructions to maximize end-to-end success across di- verse user queries on tables and unstructured documents containing domain knowledge. We evaluate on DABStep, a benchmark of 450 payment-processing questions requiring multi-step reasoning over more than nine join dimensions and wildcard joins. The setting is challenging: the original authors report only 14.55% accuracy for state-of-the-art agents. Starting from the same agent specification, we run CoCoEvolve, GEPA, and Meta-Harness for 100 iterations and compare performance on both the held-out set and the validated public leaderboard. Because evaluations are costly, CoCoEvolve uses information-gain guided observation curation (Section 2.3) to direct its budget toward likely failures with broad neighborhood reach. Appendix F.1 details this instantiation. 5 Results. Table 1 reports the train/test split for CoCoEvolve, Meta-Harness, GEPA, and the two baselines (Snowflake Cortex Code Agent, Stock Cortex Agent). On the official 450-question suite, CoCoEvolve reaches 95.24% hard accuracy, topping the validated public leaderboard; Table 15 in Appendix F gives the full comparison. Appendix F.1 specifies the scoring protocol. Train (300)Test (150) MethodEasy Hard Total Easy Hard Total Stock Cortex Agent (baseline) 57.413.820.756.08.016.0 Snowflake Cortex Code Agent 72.326.533.780.021.631.3 GEPA91.539.147.384.030.439.3 Meta-Harness87.276.778.380.078.478.7 CoCoEvolve89.496.495.384.092.891.3 Table 1: DABStep accuracy (%) on split (train: 47 easy / 253 hard; test: 25 easy / 125 hard). Analysis. Under matched splits and budgets, CoCoEvolve outperforms GEPA and Meta-Harness even when the latter is equipped with self-verification and self-testing tools that provide a compara- ble pre-commit validation surface (Appendix F.1); three mechanisms explain this gap. Live environment interaction grounds mutation in execution feedback. DABStep intentionally tests domain rules whose downstream implications must be inferred or composed across documentation and data, a failure mode emphasized by the benchmark authors (Egg et al., 2025). The live envi- ronment turns this ambiguity from guesswork into testable hypotheses. The Cortex Code mutator repeatedly formed fee interpretations, tested them through live SQL against a known merchant total, and revised its diagnosis. This live feedback exposed two hidden logic errors and enabled a veri- fied vectorized fee engine, producing the run’s largest single gain, from 28 to 95 correct answers (Appendix F.4). Curated environment tools compress otherwise lengthy investigations. The base environment ex- poses scripts, managed skills, and developer tools that encode reusable investigation and validation workflows, allowing the mutator to invoke proven procedures instead of reconstructing them across mutation cycles. During the fee-engine revision, the mutator first checked its raw SQL, then used an existing testing skill to deploy the draft as a live scratch agent and exercise multiple question families as self-validation before commit. Each passed in a single tool call, whereas the prior specification failed after roughly 40 tool calls, allowing the mutator to invest its budget in structural reasoning rather than verification plumbing (Appendix C.2). Structural changes are more robust than prompt changes. Reflective prompt evolution is effective (Agrawal et al., 2026b), but a single model call over preassembled evidence offers no path to ex- plore and validate changes to the underlying tool architecture. In a matched DABStep run, GEPA revised the instructions but left the default single-row fee calculator tool unchanged. Its evolved agent therefore aggregates 26 transaction fees through 26 tool calls and asks the LLM to sum the outputs in context, producing arithmetic drift such as 19.51 instead of 19.36. Such remedies re- quire live environment interaction: inspecting schemas, executing draft SQL, and validating outputs against known totals before commit. By testing structural edits in situ, CoCoEvolve discovered two independently verified remedies: a vectorized fee-engine stored procedure and a dynamic table that materializes per-transaction fees for downstream SQL aggregation. These solutions shift com- putation from fallible in-context arithmetic into deterministic system components, illustrating how coding-agent mutation can advance where prompt-only search plateaus (Appendix F.4). 3.2AI FUNCTIONS Modern cloud data platforms expose AI functions, declarative SQL operators that embed LLM in- ference inside query engines for row-wise tasks such as generation, classification, and extraction (Liskowski et al., 2025; Databricks, 2023; Microsoft, 2024). Industry workloads compose these operators with common table expressions (CTEs) and joins into multi-stage pipelines, so reliability depends jointly on prompts, output schemas, and pipeline topology. For instance, a pipeline may first use a generation function to identify personally identifiable information (PII) spans and define a JSON schema, then pass this plan to a downstream function that performs contextual redaction. We 6 therefore search over executable SQL programs p ∈ P , allowing CoCoEvolve to restructure both LLM prompts and operator composition for diverse row distributions, missing data, and edge cases. Setup. Each task starts from a minimal SQL program p 0 containing one generative AI-function call, which the mutator may rewrite into a multi-stage, multi-operator pipeline. Lockpicking with Interfer- ence (Appendix G.1) scores exact match on 50 labeled six-digit strings governed by a hidden ruleset. Beyond this synthetic diagnostic, we evaluate classification on Reddit comments (go emotions) and lengthy legal documents (lexglue) (Demszky et al., 2020; Chalkidis et al., 2022). We also study financial numerical reasoning: finqa requires multi-step arithmetic over tables and prose, while convfinqa distributes such reasoning across a dialogue (Chen et al., 2021; 2022). Finally, multilexnorm requires normalizing noisy, nonstandard social-media tokens into canonical forms (van der Goot et al., 2021). Because evaluation is inexpensive, this setting omits similarity-informed targeting. The mutator can instead execute candidate pipelines and repair syntax, schema, or engine-specific failures before committing, which is especially valuable for brittle SQL programs. Results. Table 2 shows that Co- CoEvolve outperforms the strongest evaluated baseline on all six tasks across classification, numer- ical reasoning, text normalization, and rule discovery. The largest gains are a 10 p improvement that exactly recovers Lockpicking’s hidden rules and a 7.7 p gain on multilexnorm, whose intermingled, context-dependent normalization patterns echo the latent-rule challenge isolated by Lockpicking. claude-sonnet-5 Method lockpicking6 goemotions finqa convfinqa lexglue multilexnorm GEPA0.8200.4250.8000.9200.7350.510 AdaEvolve0.9000.3660.7400.8850.6950.470 OpenEvolve0.5400.3750.7700.9000.6450.470 CoCoEvolve1.0000.4440.8130.9530.7370.587 Table 2: AI-function benchmark performance. Analysis. We next examine two mechanisms behind these gains. On-demand history queries help escape local minima. When a highly coupled objective plateaus, reasoning only over the current parent’s failures may not expose a route out. AdaEvolve’s adap- tive scheduler illustrates this limitation on Lockpicking (Cemri et al., 2025): it first reaches 90% at iteration 34; over the remaining 16 iterations, two stagnation-triggered paradigm generations pro- pose targeted worked examples and exhaustive condition matrices for the five residual failures, yet its best score never moves. CoCoEvolve reaches exact recovery at iteration 6, before AdaEvolve’s first paradigm generation at iteration 10, by using on-demand history tools to query evidence across lineages rather than repeatedly reframing the same residual errors (Appendix G.1). Self-validation prevents wasted mutation cycles. Without a cheap validator, mutation can waste sub- stantial search budget on SQL programs that never execute. On goemotions, GEPA repeatedly placed LATERAL FLATTEN inside a UDF; because each compiler error collapsed to a zero score without diagnostic feedback, broken variants consumed up to 71% of the budget. Alternatives such as OpenEvolve mitigate evaluation cost with cascading evaluators that stop after compilation failure, but each rejected candidate still consumes an evolutionary iteration (Sharma, 2025). CoCoEvolve instead executes drafts on a cheap sample and repairs syntax failures within the same multi-turn mutation before commit. Self-validation thus converts compiler errors from terminal scores into actionable context, reserving outer-loop evaluation for executable programs. 3.3DATA ENGINEERING Data engineering is the process of transforming raw warehouse data into trusted, queryable models. Agents often fail to construct or repair transformations due to challenges in managing the rele- vant context, such as data warehouse details, implicit relationships, and domain-specific business rules. We optimize over context-curation pipeline specifications p ∈ P that govern the creation of a context-grounded knowledge graph (KG) and a suite of executable adherence tests derived from it, both of which are supplied to a downstream agent prior to executing a transformation. We eval- uate on data-eng-bench (Snowflake AI Research, 2026), a data-engineering benchmark comprising 7 103 tasks over an enterprise warehouse with 579 tables, 19 business schemas, and more than 8,000 columns. Setup. The seed pipeline p 0 is a single agent with a minimal prompt and the scaffolding for three skills—lean human priors for authoring KGs and the adherence tests. The mutator may refine the prompt, introduce new skills, contribute to existing ones, or reorganize the pipeline into a multi-agent protocol. Φ is evaluated through a two-stage process: the mutator evolves the upstream pipeline p while the downstream agent remains fixed; a score is produced only after the agent consumes the generated KG, iterates over the corresponding adherence tests, and implements the transformation that is scored by the benchmark’s verifier. We defer further details to Appendix H. Results. As shown in Table 3, CoCoEvolve raises the solve rate on the test split from 59.1% at baseline to 77.4%, surpassing GEPA at 73.6% and Meta-Harness and AdaEvolve at 71.7%. On the dev split, CoCoEvolve reaches 68.0%, tying Meta-Harness for the highest solve rate and exceeding the baseline, AdaEvolve, and GEPA by 14.0, 8.0, and 6.0 p, respectively. Table 3: Solve rate (%) on the dev and test splits of data-eng-bench. claude-sonnet-5 DatasetBaselineMeta-HarnessGEPAAdaEvolveCoCoEvolve DevTestDevTestDevTestDevTestDevTest data-eng-bench 54.0 59.1 68.071.762.0 73.6 60.0 71.7 68.077.4 Analysis. Meta-reflection preserves globally relevant skills. During evolution, an earlier lineage in- troduced a skill to resolve a recurring dbt configuration error. A later hillclimb on an unrelated target dropped the skill. When the failure re-appeared, CoCoEvolve’s mutator retrieved the archived muta- tion plan via the ledger that first named the fix and re-implemented the skill, rather than re-deriving it from scratch. We observe that the mutator matches returning failures against past resolutions in the evolution ledger, linking current errors to the earlier mutations that fixed them. Meta-reflection thus allows for capabilities the search has already learned to remain visible to the mutator even when not directly present in the parent’s trace. 3.4MATHEMATICAL, SYSTEMS, AND PROMPT OPTIMIZATION These experiments ask whether CoCoEvolve can match specialized optimizers on established math- ematical and systems benchmarks and whether its benefits persist on HotpotQA, AIME, and IF- Bench from GEPA’s benchmark suite when live interaction is unavailable. Within each task, all arms share the task definition, initial artifact, backbone, evaluator, and data split where applicable, enabling matched end-to-end comparisons across executable programs and prompt-only systems. Setup. On math and systems tasks, each evaluator returns one scalar for the complete program. Co- CoEvolve uses a lightweight reflector to detect stagnation and decide whether to continue, request a new direction, or revert to undirected mutation. When a new direction is needed, a distinct pro- poser queries techniques from any lineage while preserving the mutator’s turn for implementation. Appendix E specifies the complete reflector-proposer protocol and prompts. AIME and IFBench use GPT-5.6-luna because Sonnet-5 saturates AIME at the seed, leaving no optimization signal. The main table omits standard deviations for readability; bold and underlining mark the best and second- best results within each backbone, with ties sharing marks. Appendix I reports standard deviations, while Appendix J.1 details the prompt protocol. Results. Table 4 reports the mathematical and ADRS systems benchmarks. CoCoEvolve is best or tied best in 25 of 44 Avg/Best cells, including 13 of 24 math and 12 of 20 ADRS cells, more than any other method. On Sonnet-5, the largest gains occur on plateau-prone Signal Processing, TXN, and Heilbronn Convex; on Signal Processing, CoCoEvolve reaches 0.766 versus AdaEvolve’s 0.747. Gaps largely vanish on saturated objectives, while mixed GPT-5.6-terra results show that the advantage depends on the backbone. Appendix I.3 analyzes plateau escape, structural family switch- ing, and saturation controls task by task. Table 5 reports prompt optimization, where CoCoEvolve leads GEPA and the seed on every HotpotQA and IFBench split, while AIME is inconsistent under repeated evaluation. 8 Mathematical OptimizationADRS Systems Circle Packing (Square) Circle Packing (Rect) Heilbronn (Triangle) Heilbronn (Convex) MinMax Distance Signal ProcessingCloudcast (↓)EPLBPrismLLM-SQLTXN (N = 26)(N = 21)(N = 11)(N = 13)(d = 3,N = 14) StrategyAvgBestAvgBestAvgBestAvgBestAvgBestAvgBestAvgBestAvgBestAvgBestAvgBestAvgBest Human/SOTA–2.636–2.366–0.037–0.031–0.240–0.718–626.200–0.127–21.890–0.692–2,725 Backbone: Sonnet-5 OpenEvolve2.6362.6362.3662.3660.0320.0350.0280.0290.2380.2400.6500.673660.693 660.689 0.127 0.127 26.236 26.2560.713 0.726 3,945 4,016 GEPA2.6022.6362.3652.3660.0360.0370.0300.0310.2400.2400.6030.631621.090 618.0800.133 0.14526.25626.2560.524 0.752 3,874 4,000 AdaEvolve2.6332.6362.3642.3660.0350.0370.0290.0310.2400.2400.7470.766618.097618.088 0.1450.145 26.391 26.660 0.7240.7334,1154,132 CoCoEvolve2.6352.6362.3632.3660.0350.0370.0310.0310.2400.2400.7660.784618.063 618.060 0.145 0.146 26.25626.2560.728 0.732 4,292 4,348 Backbone: GPT-5.6-terra OpenEvolve2.6362.6362.3652.3660.0300.0310.0250.0270.2400.2400.5650.608618.096 618.093 0.127 0.127 26.076 26.152 0.7240.7304,107 4,184 GEPA2.634 2.6362.3642.3660.0350.0350.0260.0280.2400.2400.6330.661618.081 618.070 0.127 0.127 26.25926.2640.719 0.724 4,268 4,348 AdaEvolve2.6332.6342.3662.3660.0350.0350.0240.0240.2400.2400.7920.813618.085 618.070 0.1420.145 25.674 25.734 0.717 0.724 4,144 4,167 CoCoEvolve2.6342.6362.3642.3660.0340.0370.0250.0270.2400.2400.7560.805618.082618.0710.145 0.14526.338 26.501 0.728 0.730 4,2504,329 Table 4: Mathematical and ADRS systems benchmarks. Higher is better except for Cloudcast. Human/SOTA is the maximum published human or automated result reported by AlphaEvolve and AdaEvolve. Full-precision means, standard deviations, and best values appear in Tables 18 and 19. HotpotQA (Sonnet-5)AIME (luna)IFBench (luna) MethodTrainValTestTrainValTestTrainValTest Baseline (unmodified seed)0.3930.3130.363 0.919± 0.056 0.882± 0.0260.833± 0.000 † 0.902± 0.0030.864± 0.0030.663± 0.027 GEPA0.7130.7000.683 0.919± 0.0780.830± 0.0560.833± 0.0580.891± 0.0180.886± 0.0090.683± 0.005 CoCoEvolve0.8000.7830.793 0.919± 0.026 0.882± 0.046 0.878± 0.051 0.927± 0.010 0.921± 0.009 0.705± 0.024 Table 5: Prompt optimization on HotpotQA, AIME, and IFBench. HotpotQA reports a single Sonnet-5 run; AIME and IFBench report mean ± standard deviation over three evaluations of each final candidate using openai-gpt-5.6-luna; these repeats measure evaluation noise, not search variance. Bold marks column-best results that are robust to measurement noise; † the ob- served zero deviation is coincidental. Analysis.Across the mathematical and systems benchmarks, CoCoEvolve recovers the gains of AdaEvolve’s specialized three-level adaptive controller through meta-reflection over the search history, offering a more flexible route to stagnation-aware search. The strategy ledger turns plateaus into new search directions. On Signal Processing, the global best remained at 0.7534 for 26 iterations even though the current parent and champion represented distinct Savitzky-Golay and Butterworth filter families. Reasoning over the strategy ledger, the lightweight reflector recognized a population-wide plateau and requested a new proposal rather than another local mutation or crossover. Using on-demand history queries to identify what the search had not tried, the separate proposer found an orthogonal family: an L1 trend filter formulated as a linear program. The mutator implemented this paradigm and raised the score to 0.7745 in one iter- ation. This trace illustrates how the strategy ledger and on-demand history tools separate stagnation detection, paradigm selection, and implementation: the search history reveals not only which ideas succeeded, but which structural alternatives remain unexplored. On-demand history queries improve prompt optimization even without live environment interaction. On HotpotQA, CoCoEvolve reaches 0.793 held-out accuracy versus GEPA’s 0.683 and the seed’s 0.363, an 11 p margin in GEPA’s own setting. IFBench preserves the ordering CoCoEvolve > GEPA > seed on every repeat and split, although the held-out margin is narrower. AIME provides no reliable separation: all arms lie in the same 0.83–0.88 test band, while 17–27% of individual answers change correctness across repeated evaluations of an unchanged prompt. Together, these results isolate the value of on-demand history access for text-only artifacts, while AIME illustrates how stochastic evaluation can erase apparent search gains. 4RELATED WORK Evolutionary Harnesses.LLM-guided evolutionary optimization shifts improvement from training-time parameter updates toward inference-time search, using LLMs as semantic mutation operators over prompts, algorithms, and programs. Building on quality-diversity search (Mouret & Clune, 2015), systems such as FunSearch, AlphaEvolve, and OpenEvolve apply this paradigm to scientific and algorithmic discovery (Romera-Paredes et al., 2024; Novikov et al., 2025; Sharma, 2025). GEPA enriches mutation with language-valued execution traces that expose reasoning, tool calls, outputs, and errors beyond scalar rewards (Agrawal et al., 2026b). Subsequent work broad- ens the optimized artifact space through unified declarative APIs (Agrawal et al., 2026a), allocates 9 search adaptively (Cemri et al., 2025; Lange et al., 2025), and standardizes outer harnesses (Ursekar et al., 2026). For prompts and LM pipelines, EvoPrompt and Promptbreeder evolve prompts with LLM-driven operators (Guo et al., 2025; Fernando et al., 2024), MIPROv2 jointly searches instruc- tions and demonstrations for multi-stage programs (Opsahl-Ong et al., 2024), and TextGrad back- propagates natural-language feedback through compound systems (Yuksekgonul et al., 2025). These methods generally retain a parent-local, one-shot mutation interface in which the harness preselects evidence and evaluates the proposed edit only after generation. Promptbreeder also evolves its mu- tation prompts, whereas CoCoEvolve changes the mutation interface itself through meta-reflection over the search history and target environment. Meta-Harness (Lee et al., 2026) similarly uses a coding-agent proposer with filesystem access to prior traces; CoCoEvolve combines compact cu- rated context with structured, on-demand queries over evolutionary history and a live environment. Context Curation Approaches. The efficacy of LLM agents relies heavily on rich, domain-specific context, such as organizational schemas (D ̈ ohmen et al., 2026) or dynamically constructed failure taxonomies (Cojocaru et al., 2026). Recent frameworks automate context engineering by treating contexts as evolving playbooks (Zhang et al., 2026) or co-evolving curation skills alongside artifacts (Ye et al., 2026). CoCoEvolve applies this principle to the optimization trajectory itself, distilling accumulated search history into a compact ledger. Limitations of Evolutionary Approaches. AI-driven evolutionary frameworks exhibit systematic vulnerabilities, including hidden regressions (Zhou et al., 2026), overfitting to the initial task set under continual learning (Wang et al., 2026), and context costs that grow with the accumulated archive (Lee et al., 2026). CoCoEvolve addresses these failure modes with bounded, queryable access to historical co-regressions, helping the optimizer avoid cycling through brittle solutions that break previously solved cases. 5CONCLUSION We present CoCoEvolve, an evolutionary optimizer for production AI systems. By both probing the environment and querying an evolutionary ledger and prior lineages, CoCoEvolve’s mutator extends parent-local reflection into meta-reflection over the search process, enabling it to diagnose failures, reconstruct hidden mechanisms, preserve complementary discoveries, and verify structural revisions before committing. Across domains, our results suggest that the combination of curating selected historical evidence, providing on-demand tools for probing the live environment and evolutionary history, and using a coding-agent mutator to reason over both offers an effective path to improving the prompts, code, tools, data models, and pipelines of real-world AI systems. ACKNOWLEDGMENTS We thank Danmei Xu, Garett Tok, Zihan Li, Varich Boonsanong, Kyle Schmaus, and Boxin Jiang for insightful discussions, feedback and suggesting relevant datasets. REFERENCES Lakshya A. Agrawal, Donghyun Lee, Shangyin Tan, Wenjie Ma, Karim Elmaaroufi, Rohit San- dadi, Sanjit A. Seshia, Koushik Sen, Dan Klein, Ion Stoica, Joseph E. Gonzalez, Omar Khattab, Alexandros G. Dimakis, and Matei Zaharia. optimize anything: A universal API for optimizing any text parameter. In ACM Conference on AI and Agentic Systems (CAIS ’26), 2026a. URL https://arxiv.org/abs/2605.19633. Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflective prompt evolution can outperform reinforcement learning. In The Fourteenth International Con- ference on Learning Representations, 2026b. URL https://openreview.net/forum?i d=RQm2KQTM5r. Oral. Amazon Web Services. Agents for Amazon Bedrock. https://aws.amazon.com/bedrock /agents/, 2023. Accessed: 2026-08-09. 10 Mert Cemri, Shubham Agrawal, Akshat Gupta, Shu Liu, Audrey Cheng, Qiuyang Mang, Ashwin Naren, Lutfi Eren Erdogan, Koushik Sen, Matei Zaharia, Alex Dimakis, and Ion Stoica. AdaE- volve: Adaptive LLM driven zeroth-order optimization. arXiv preprint arXiv:2602.20133, 2025. URL https://arxiv.org/abs/2602.20133. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Mar- tin Katz, and Nikolaos Aletras. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (ACL), p. 4310–4330, 2022. URL https://aclanthology.org/2022.acl-l ong.297/. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 3697–3711, 2021. URL https://aclanthology.org/2021.emnlp-main.300/. Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question an- swering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 6279–6292, 2022. URL https://aclanthology.org/2022. emnlp-main.421/. Andrei Cojocaru, Mert Cemri, Melissa Pan, Shu Liu, Shubham Agarwal, Alex Krentsel, Jay Tang, Kannan Ramchandran, Joseph Gonzalez, Matei Zaharia, Alex Dimakis, and Ion Stoica. Fantastic adaptive taxonomies and how to use them. In FAGEN Workshop at ICML 2026, 2026. URL https://openreview.net/pdf?id=krDmXYWzmJ. Best Paper Award. Databricks. Introducing AI functions: Integrating large language models with Databricks SQL. https://w.databricks.com/blog/2023/04/18/introducing-ai-funct ions-integrating-large-language-models-databricks-sql.html, 2023. Accessed: 2026-08-09. Databricks. Mosaic AI agent framework. https://w.databricks.com/product/mac hine-learning/mosaic-ai-agent-framework, 2024. Accessed: 2026-08-09. dbt Labs. What is dbt? dbt Developer Hub, 2026. URL https://docs.getdbt.com/docs /introduction. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), p. 4040–4054, 2020. URL https://aclanthology.org/2020.acl-main.372/. Till D ̈ ohmen, Jacob Matson, and Jordan Tigani. Towards a context layer for self-improving data agents. In SAO Workshop at ACM Conference on AI and Agentic Systems (CAIS ’26), 2026. URL https://bauplanlabs.github.io/SAO-workshop/papers/73.pdf. Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, and Thomas Wolf.DABstep: Data agent benchmark for multi-step reasoning.arXiv preprint arXiv:2506.23719, 2025. URL https://arxiv.org/abs/2506.23719. Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt ̈ aschel. Promptbreeder: Self-referential self-improvement via prompt evolution. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 13481–13544, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/fernando24a.html. Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Evoprompt: Connecting llms with evolutionary algorithms yields powerful prompt optimizers. In Proceedings of the 31st International Conference on Computational Linguistics, 2025. URL https://aclanthology.org/2025.coling-main.345/. 11 Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin. ShinkaEvolve: Towards open-ended and sample-efficient program evolution. arXiv preprint arXiv:2509.19349, 2025. URL https: //arxiv.org/abs/2509.19349. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta- Harness: End-to-end optimization of model harnesses. In Conference on Language Modeling (COLM), 2026. URL https://arxiv.org/abs/2603.28052. Paweł Liskowski, Benjamin Han, Paritosh Aggarwal, Bowei Chen, Boxin Jiang, Nitish Jindal, Zihan Li, Aaron Lin, Kyle Schmaus, Jay Tayade, Weicheng Zhao, Anupam Datta, Nathan Wiegand, and Dimitris Tsirogiannis. Cortex AISQL: A production SQL engine for unstructured data, 2025. URL https://arxiv.org/abs/2511.07663. Microsoft. Invoke AI capabilities using Azure SQL database and Azure OpenAI. https://lear n.microsoft.com/en-us/azure/azure-sql/database/invoke-ai-with-a zure-openai, 2024. Accessed: 2026-08-09. Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909, 2015. URL https://arxiv.org/abs/1504.04909. E. A. Nadaraya. On estimating regression. Theory of Probability and Its Applications, 9(1):141–142, 1964. doi: 10.1137/1109020. URL https://doi.org/10.1137/1109020. Alexander Novikov, Ng ˆ an Vu, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. AlphaEvolve: A coding agent for scientific and algorithmic discovery. White paper, Google DeepMind, 2025. URL https://storage.go ogleapis.com/deepmind-media/DeepMind.com/Blog/alphaevolve-a-gem ini-powered-coding-agent-for-designing-advanced-algorithms/Alph aEvolve.pdf. Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 9340–9366, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/20 24.emnlp-main.525. URL https://aclanthology.org/2024.emnlp-main.525/. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nature, 625:468–475, 2024. doi: 10.1038/s41586-023-069 24-6. Asankhaya Sharma. OpenEvolve: Open-source implementation of AlphaEvolve. GitHub, 2025. URL https://github.com/codelion/openevolve. Snowflake AI Research. Data-eng-bench: A data engineering benchmark for AI agents. Snowflake Engineering Blog, 2026. URL https://w.snowflake.com/en/blog/engineerin g/data-eng-bench-data-engineering-agent-benchmark/. Snowflake Inc. Cortex agents. Snowflake Documentation, 2026a. URL https://docs.snowf lake.com/en/user-guide/snowflake-cortex/cortex-agents. Snowflake Inc. Snowflake CoCo: Snowflake-native AI coding agent for data. Snowflake Product Page, 2026b. URL https://w.snowflake.com/en/product/snowflake-coc o/. Varun Ursekar, Apaar Shanker, Veronica Chatrath, Yuan Xue, and Samuel Marc Denton. VeRO: A harness for agents to optimize agents. In ICML 2026, 2026. URL https://arxiv.org/ab s/2602.22480. 12 Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Barbara Plank, Benjamin Muller, I ̃ naki San Vi- cente Roncal, Nikola Ljube ˇ si ́ c, ̈ Ozlem C ̧ etino ̆ glu, Rahmad Mahendra, Talha C ̧ olako ̆ glu, Timothy Baldwin, Tommaso Caselli, and Wladimir Sidorenko. MultiLexNorm: A shared task on multi- lingual lexical normalization. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT), 2021. URL https://aclanthology.org/2021.wnut-1.55/. Wenxiao Wang, Priyatham Kattakinda, and Soheil Feizi. Do agent optimizers compound? a continual-learning evaluation on Terminal-Bench 2.0. arXiv preprint arXiv:2607.14004, 2026. URL https://arxiv.org/abs/2607.14004. Geoffrey S. Watson. Smooth regression analysis. Sankhy ̄ a: The Indian Journal of Statistics, Series A, 26(4):359–372, 1964. URL https://w.jstor.org/stable/25049340. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 2369–2380, 2018. URL https://aclanthology.org/D18-1 259/. Haoran Ye, Xuning He, Vincent Arak, Haonan Dong, and Guojie Song. Meta context engineering via agentic skill evolution. arXiv preprint arXiv:2601.21557, 2026. URL https://arxiv. org/abs/2601.21557. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative AI by backpropagating language model feedback. Nature, 639:609–616, 2025. doi: 10.1038/s41586-025-08661-4. Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kama- nuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. ACE: Agentic context engineering: Evolving contexts for self-improving language models. arXiv preprint arXiv:2510.04618, 2026. URL https://arxiv.org/abs/2510 .04618. Yajie Zhou, Ao Li, Ashwin Silla, Zaoxing Liu, and Vyas Sekar. AIChilles: Automatically uncov- ering hidden weaknesses in AI-evolved systems. arXiv preprint arXiv:2606.15834, 2026. URL https://arxiv.org/abs/2606.15834. ATABLE OF CONTENTS This appendix is organized as follows. Appendix C specifies the evolutionary loop (Appendix C.1), the history-query and workspace tools (Appendix C.2), the similarity-graph InfoGain branch used by agent evolution (Appendix C.3), the experimental hyperparameters (Appendix C.4), and the bud- geted ℓ p roster (Appendix C.5). Appendix D reports parent selection, mutation and self-validation, history access, and similarity-manifold targeting ablations. The history-access study includes Sig- nal Processing and DABStep. Each ablation subsection reports its trial count and score provenance. Appendix E records the mutator, reflector, and proposer prompts. Appendix F gives the DABStep protocol, analyzes Meta-Harness’s post-improvement churn, and presents an evolved agent instruc- tion. Appendix G presents the Lockpicking synthetic diagnostic (Appendix G.1) and describes the AI-function experiments and their evolved artifacts. Appendix H describes the data-engineering pipeline, its evaluation, and an evolved artifact. Appendix I records math and ADRS evaluator set- tings and the detailed result tables (Appendix I.2). Appendix I.3 gives the per-task analysis: which methods lead each cell, then Heilbronn Convex, Circle Packing, Cloudcast, and TXN traces (Fig- ures 4–7). Appendix J gives the methodology for all three prompt-optimization tasks (Appendix J.1) and a sim-graph-targeted, cascaded-evaluation efficiency variant of HotpotQA (Appendix J.2). Ap- pendix K, placed last, includes the complete base and final programs for Heilbronn Convex, Cloud- cast, and TXN. 13 BUSE OF LARGE LANGUAGE MODELS (LLMS) AND OTHER AI TOOLS The authors used large language models (LLMs) to assist in literature discovery and polishing the manuscript’s prose. The authors reviewed and verified all suggested references, claims, and revi- sions, and assume full responsibility for the final content. Coding assistants, including Snowflake Cortex Code, Anthropic Claude, and Cursor, helped implement the authors’ ideas. The methodol- ogy, scientific contributions, and core results are the authors’ original work. CALGORITHM AND METHOD This appendix instantiates the operators of Section 2. Section C.1 gives the evolutionary loop; Section C.2 lists the history tools added through Augment and the domain tools supplied by E 0 ; Section C.3 records the kernel, decay, and softmax used for curated observations; Section C.4 lists the hyperparameters used in the experiments of Section 3; and Section C.5 specifies the budgeted ℓ p roster used as a population objective and as a parent-selection policy. C.1EVOLUTIONARY LOOP Algorithm 1 CoCoEvolve evolutionary loop. Require: seed p 0 , budget T , evaluator f , parent policy π, base environmentE 0 ⊆E 1: A 0 ←p 0 ,H 0 ←∅ 2: for t = 1,...,T do 3: p ′ t ∼ π(A t−1 )▷ Parent selection 4: c t ← Curate(H t−1 )▷ Context curation 5: E t ← Augment(E 0 ;p ′ t ,H t−1 )▷ Environment augmentation 6: p t ∼ MetaReflect(p ′ t ,c t ;E t )▷ Agentic investigation 7:selectT t ⊆T ; evaluate p t with f to obtain S t and O t 8: H t ←H t−1 ∪(p t ,T t ,S t ,O t ) 9:updateA t−1 with p t to formA t 10: end for 11: return arg max p∈A T Φ(p) The loop selects parent p ′ t from A t−1 using π; the parent policy uses either MAP-Elites or the ℓ p roster of Section C.5. Context curation prepares c t , while environment augmentation adds on- demand history-query tools toE 0 to formE t . Depending on the task, Curate reflects on the strategy ledger and selects a target instance using the similarity graph of Section C.3. Because operational requirements vary by domain, these preparation mechanisms are layered according to the search objective. We detail the exact workflow applied for each experiment in its task subsection. In general, the injected bundle c t contains the approach ledger (recent strategies and their score deltas), the active paradigm text if one is in force, and island elites. When an instance pool is defined, c t also includes the curated target observation and any donor program that has already solved it. The augmented environment E t combines the domain tools in E 0 with the opt-in history tools of Section C.2. The Cortex Code coding-agent mutator then performs MetaReflect by investigating, implementing, and validating the child, after which admission is typically a MAP-Elites update on a complexity–diversity grid. C.2ON-DEMAND HISTORY AND ENVIRONMENT TOOLS Table 6 lists the on-demand tools provisioned inE t . The mutator can call these tools from its task workspace while editing and validating a candidate. The ledger is injected every iteration; tool re- sults enter the context only when requested, so the full archive need not appear in the prompt. For archive access, the mutator calls python3 evo.py to query prior programs, lineages, scores, and traces. For local execution, the constrained evo sweep.py runner executes the candidate with- out granting unrestricted command execution. The math, ADRS, Signal Processing, and DABStep experiments all expose lineage and trajectory queries. Per-task queries are enabled for DABStep and Signal Processing, whose evaluators expose instance-level or per-signal outcomes, and disabled 14 for the remaining math and ADRS tasks, which return only a scalar score. DABStep additionally overlays warehouse tools that issue SQL, deploy a scratch agent, and pre-screen a draft against a target question before committing the child. ChannelToolReturns Lineage getprogramcode body of a program getparent / getlineageparent id; children with score deltas getbestprogramstop-N programs by primary score getdiverseprogramsbest program per occupied MAP-Elites cell getarchitecturehistoryper-cell landing history Trajectory getrecentscoresrecent primary-score trace getthinkingtracetool/thinking events of iteration I Per-task getquestionsolverstop programs on one instance, with parent getquestionmemorychild<parent regressions on one instance getsuggestedquestionsinstances ranked worst-first for a program gethardquestionsinstances by population-best score, ascending Warehouse (DABStep) queryread-only SQL against live tables executesqlallowed write SQL against the warehouse testagent / testdraftscore a scratch agent on question Q checkdeploycompile-check one CREATE DDL analyst querytext-to-SQL against the semantic view Table 6: On-demand tools available inE t . Lineage, trajectory, and per-task tools are added through Augment to query the evolutionary archive; the DABStep base environment E 0 additionally pro- vides live-warehouse validation tools. C.3CURATED OBSERVATIONS: KERNEL, DECAY, AND SOFTMAX Section 2.3 writes a generic similarity kernel κ. This section applies to the DABStep agent-evolution setting, which evaluates a program on many instances and maintains a similarity graph over that pool. Math, ADRS, and Signal Processing have no such pool and skip this channel, even though Signal Processing still exposes per-signal query tools (Table 7). We precompute an embedding- similarity matrix κ on V , with values in [0, 1]. DABStep specializes the estimator in Section 2.3 by replacing κ(τ,τ ′ ) with κ(τ,τ ′ ) 4 in both the numerator and denominator, giving more weight to nearby tested instances. After each mutation, a local success or regression update refines the esti- mate. Estimates for untested coordinates are capped at 0.7, and coordinates with observed failures are floored at 0.1. The next targetτ ∗ is drawn by softmax over gain with temperature 0.1. Candidates exclude instances the parent has already solved (score≥ 0.9). The decay in Section 2.3 is instantiated as δ(τ ) = 1if any program inA t has solved τ, (1/2) c(τ) otherwise, where c(τ ) is the number of times τ has been selected as the mutation target without success. A solve anywhere in the archive therefore resets the penalty, so a newly acquired donor can be reused; repeated failures on an unsolved instance geometrically suppress it. C.4HYPERPARAMETERS Table 7 records the settings used in Section 3. The default math, ADRS, and DABStep runs sam- ple parents from MAP-Elites islands using exploration, elite selection, and fitness-weighted draws. Math and ADRS use exploration weight 0.2 and elite-selection weight 0.65. The AI-function ob- jective and the parent-selection ablation instead use the ℓ p roster of Section C.5. InfoGain targeting (Section C.3) is on for DABStep only. Signal Processing exposes per-task query tools because its evaluator returns per-signal scores, al- though it has no V and does not use InfoGain. AI functions use the ℓ p roster and full validation evaluation rather than InfoGain targeting; they are omitted from the table because their remaining settings follow Section 3.2. 15 Math / ADRSSignal ProcessingDABStep Iteration budget T100100100 Islands / population2 / 502 / 801 / 40 InfoGain targetingoffoffon Per-task query toolsoffonon Task workspace + history queriesononon Warehouse / scratch-agent toolsoffoffon Cascade thresholdstask-dependent ∗ [0.3, 0.6][0.005] LLM temperature0.7 (framework default)0.60.7 Table 7: Domain hyperparameters and tool availability. All listed domains edit candidates in a task workspace and expose on-demand history queries; DABStep additionally enables InfoGain targeting and warehouse tools. ∗ Cascade evaluation is enabled for Circle Packing (Square) with thresholds [0.5, 0.8] and disabled for the remaining math and ADRS tasks. C.5BUDGETED POPULATION ℓ p OBJECTIVE AND PARENT SELECTION Let A t denote the archive after iteration t. Suppose each program p ∈ A t is evaluated along N dimensions, with per-dimension score s i (p). A dimension may represent a task instance, objective, or behavioral criterion. We associate each program with the set I(p) =i∈ [N ] : s i (p)≥ θ i of dimensions on which it is considered successful. The thresholds θ i may be absolute or deter- mined relative to the current population. For binary per-dimension scores, I(p) is exactly the set of dimensions solved by the program. For continuous scores, we use either a calibrated absolute threshold or a per-dimension population quantile, allowing each dimension to be compared against its own achievable score range. Budgeted diversity roster. AlthoughH t retains the complete evaluation history, only a budgeted active roster R t ⊆ A t , |R t | ≤ K, is used to represent the population and select the next parent. Starting withR (0) t =∅ and C (0) =∅, we greedily choose p j = argmax p∈A t (j−1) t |I(p)\ C (j−1) |, define its marginal contribution as m j =|I(p j )\ C (j−1) |, and update R (j) t =R (j−1) t ∪p j , C (j) = C (j−1) ∪I(p j ). Construction stops after K programs or when no remaining program contributes new coverage. This is the standard greedy maximum-coverage construction: it retains programs because of capabilities not already represented by higher-ranked roster members, rather than solely because of aggregate fitness. Population ℓ p score. Given the ordered roster marginals m = (m 1 ,...,m |R t | ), we define L p (R t ) = ∥m∥ p N = 1 N   X j m p j   1/p . Equivalently, the evaluator may report its p-th power J p (R t ) = L p (R t ) p = P j m p j N p , which induces the same ordering for fixed p. At p = 1, the score is the fraction of dimensions cov- ered by the roster. For p > 1, it also rewards consolidation: for the same total coverage, the score is higher when fewer programs carry that coverage. Greedy roster construction retains complementary capabilities, while the objective rewards programs that combine them. 16 ℓ p as an optimization objective. For tasks in which population quality is itself the target, a newly evaluated child p t is assessed by inserting it into the candidate pool, rebuilding the roster, and mea- suring ∆J p = J p (R t |A t ∪p t )− J p (R t |A t ). The evolutionary loop then directly optimizes J p . This is the formulation used in our AI-function setting with p = 2, where J 2 = P j m 2 j /N 2 rewards both broad population coverage and the consolidation of specialist capabilities. ℓ p as a parent-selection policy. When the task retains its original scalar objective, the same roster can instead be used only to choose the next parent. We assign each roster member probability Pr(p j |R t ) = m p j P ℓ m p ℓ . This samples programs according to their contribution to the population ℓ p score without replacing the task’s evaluator. At p = 0, the policy is uniform over roster members; at p = 1, it is proportional to exclusive marginal coverage; our default p = 2 emphasizes large complementary contributions quadratically; and as p → ∞, it concentrates on a maximum-marginal roster member. An agentic variant exposes the same roster, marginals, and population score to an LLM and allows it to choose the parent directly, using the mechanical distribution as a fallback. Table 9 reports a HotpotQA sweep of p∈0.5, 1, 1.5, 2 for this parent-selection policy. The lp arm uses p = 2. C.5.1RARE-CAPABILITY CARRIER SELECTION Marginal coverage alone can under-allocate mutations to valuable specialists. Suppose most pro- grams solve the same large set of easy dimensions, while different programs solve only one or two distinct hard dimensions. The greedy roster retains these specialists, but for p > 1 its mechani- cal distribution can place nearly all probability on the first member, whose marginal includes the common easy dimensions. To remove this dependence on raw instance frequency, let ρ i = 1 |A t | X p∈A t I[i∈I(p)], h i = (1− ρ i ) γ , where ρ i is the population solve rate and γ > 0 controls how strongly rare dimensions are empha- sized. We define the hard-capability mass carried by a program as H(p) = N X i=1 h i I[i∈I(p)]. We incorporate rare-capability mass into parent selection by sampling according to q rare (p)∝ exp β h λ e Φ(p) + (1− λ)H(p) p i , where e Φ is normalized aggregate fitness. The fitness term favors the strongest carrier of a given rare capability, rather than weak programs in general, while H(p) p gives increasing preference to lineages that accumulate multiple hard capabilities. If all specialists carry one equally rare capability and have equal fitness, they remain indistinguishable and are sampled symmetrically; once one lineage combines multiple rare capabilities, its selection probability increases. Because no selector can exploit a dimension that no current program solves, this policy can be mixed with the baseline selector, q(p) = (1− ε)q rare (p) + εq base (p). The rare-capability term is part of the parent-selection mechanism. It changes which existing pro- gram is selected for mutation without adding a donor or changing the evaluation pool. All candidates are evaluated on the same instances, so their capabilities remain directly comparable. DABLATION DETAILS This section reports four ablations against the mechanisms of Appendix C and Section 3. Par- ent selection (Appendix D.1) compares four parent-selection setups and examines the ℓ p sampling exponent. Three claim-driven ablations each isolate one further axis of the evolutionary loop: muta- tion execution and self-validation (Appendix D.2), the history-access interface (Appendix D.3), and similarity-manifold targeting (Appendix D.4). 17 Common controls. Each claim-driven ablation below estimates the effect of one named treatment while holding the rest of the loop fixed; a comparison that changes two axes at once is not reported as a causal ablation. Unless stated as the named treatment, every arm of a given ablation shares the same backbone model, temperature, and model-facing mutation budget; the same seed program and train/test split; the same parent-selection policy and population configuration; the same editable artifact surface; the same number of outer-loop iterations; and the same evaluator thresholds and per-child evaluation allowance. Persistent per-program and archive-level solved-question state is retained across arms, so a no-graph arm does not remember only the current minibatch. The tables below report the final metrics available for each ablation. Each subsection states the number of trials. These controls are already in force below unless a subsection states an exception. D.1PARENT SELECTION We test whether useful parent selection requires both preserving complementary capabilities and directing mutations toward programs that can consolidate them. The four-way comparison examines which programs remain eligible as parents; the HotpotQA exponent sweep probes how mutation effort is allocated among them. Setup. We implement four parent-selection policies in the same OpenEvolve codebase. Within each task, all arms use the same seed, split, backbone, mutation process, island configuration, and 100-iteration budget. Only the parent-selection policy changes. select-best always chooses the current best program. map-elites uses OpenEvolve’s exploration/exploitation/fitness draw from the MAP-Elites grid. pareto-frontier samples the per-instance Pareto frontier by win count, as in GEPA. lp builds a greedy K-roster from per-instance solved sets and samples member j with probability proportional to m p j , where m j is its marginal coverage (Appendix C.5); here p = 2. The frontier and roster draw from all islands. Table 8 compares the complete setups on each task. To probe whether stronger preference for large marginal contributions helps the roster consolidate specialists, we compare HotpotQA configurations with p∈0.5, 1, 1.5, 2 while keeping the roster construction fixed. All four use openai-gpt-5.6-luna, a 3-example mutation minibatch from train-150, val-300 exact match for search and export, and one 100-iteration search per setting. The p = 2 run reuses the HotpotQA lp result above; p is the only treatment varied in this comparison. Results. Table 8 shows the results for the four parent-selection setups and Table 9 shows the results for the HotpotQA ℓ p sweep. HotpotQAIFBench Parent selectionTest EM∆Test score∆ select-best0.6400 −0.02670.6592 −0.0360 map-elites0.666700.69520 pareto-frontier0.6433 −0.02340.6918 −0.0034 lp (p = 2)0.6933+0.02660.7003+0.0051 Table 8: Parent-selection setups on HotpotQA and IFBench. One 100-iteration search per arm; entries are single held-out test evaluations. HotpotQA uses exact match; IFBench uses its per- constraint score. Each ∆ is the score difference from OpenEvolve’s default map-elites arm on the same task. The lp row uses p = 2. Bold marks the highest observed score in each task. Roster exponentValidation EMTest EM∆ test p = 0.50.60330.5867 −0.0733 p = 10.71000.66000 p = 1.50.59670.5700 −0.0900 p = 20.73000.6933+0.0333 Table 9: HotpotQA validation and held-out test exact match across ℓ p roster exponents. ∆ is the test-EM difference from p = 1; bold marks the highest observed score. 18 Analysis. These selectors use different evidence to identify useful parents. select-best spends every mutation on the current incumbent, leaving complementary capabilities in other pro- grams unused. map-elites preserves programs in distinct grid cells, but those cells need not correspond to distinct solved instances. pareto-frontier uses per-instance wins, yet win- count weighting does not discount wins already represented by another parent. The greedy ℓ p roster instead retains a program only while it adds uncovered instances, then favors roster members with larger exclusive contributions. The ℓ p roster has the highest observed held-out score on both tasks in Table 8, consistent with allocating mutations by complementary coverage rather than grid diversity alone. The exponent sweep probes how to allocate mutations after complementary parents have entered the roster. Flatter weighting spends more turns on small-marginal specialists; quadratic weighting concentrates ordinary mutations on parents that already contribute many distinct solves, giving later edits more opportunities to consolidate those capabilities. The p = 2 row leads on both validation and held-out test, consistent with a balance between consolidation pressure and specialist access. The dip at p = 1.5 shows that the effect is not simply monotonic in p: changing the weighting changes which lineages receive repeated edits, and those search paths can produce different final programs. D.2MUTATION EXECUTION AND SELF-VALIDATION This ablation separates two benefits of an interactive mutator: editing the current candidate with access to its environment, and testing that candidate before committing it. The first can support broad instruction-level fixes; the second is especially useful for changes to executable code or dependent objects. Setup. We run this ablation on Heilbronn Convex (N =13) and DABStep. Within each task, the three arms share the parent policy, population and island settings, feature grid, history-access in- terface, iteration budget, targeting configuration, and reflector/proposer configuration. They differ in the mutator’s action space. Heilbronn Convex has no instance similarity graph; DABStep keeps its similarity graph enabled across all three arms. The reflector/proposer remains on in every arm, matching the published configurations. A1 (static mutator) is one LLM completion over the parent, ledger, and active paradigm; it cannot inspect the workspace or execute anything. A2 (agentic, no candidate testing) can edit the candidate and query history through evo.py, but cannot execute the edited candidate before committing it. A3 (full mutator, published) adds pre-commit execution: a local Python sweep on Heilbronn Convex, or deployment and end-to-end testing of the edited agent on DABStep. On DABStep, A1 can only diagnose from injected artifacts, A2 can additionally query the database but cannot deploy the edited agent, and A3 has the complete query, deployment, and testing interface. Results. We report three Sonnet-5 trials per Heilbronn Convex arm. One trial stopped after 83 iterations; the others ran for 100. Its score is included in the reported summary, so that arm’s mean is not based on three equal-budget trials. One 100-iteration trial is reported per arm on DABStep. All DABStep Train/Test values are fresh full-pool re-evaluations of each arm’s champion rather than in-run estimates based on limited evaluations; A1’s initial estimate used a 25-question bootstrap sample. A1 never advanced past the first evaluation gate, so its champion is the unmodified seed also reported in Table 1 and as A1 in Table 13. A3 on both tasks reuses the published champion rather than a new run; the DABStep champion is also the C3 arm in Table 13. Each ∆ is the raw-count difference from A1. Heilbronn Convex (min triangle area / hull area)DABStep ArmMean± StdBestTrain (of 300)∆Test (of 150)∆ A1 static mutator0.0260± 0.00030.0266101–47– A2 agentic, no candidate test0.0270± 0.00020.0276202+10198+51 A3 full mutator (CoCoEvolve)0.03066± 0.000360.03094286+185137+90 Table 10: Mutation-execution ablation on Heilbronn Convex and DABStep. Values summarize three trials per arm for Heilbronn Convex and one full-pool champion evaluation per arm for DABStep; ∆ is relative to A1. 19 Analysis. Performance improves monotonically from the static mutator (A1), to agentic editing without pre-commit candidate testing (A2), to the full mutator (A3) on both tasks. The mean Heil- bronn Convex score rises from 0.0260 to 0.0270 to 0.03066, while DABStep test accuracy rises from 47/150 to 98/150 to 137/150. The two transitions add different capabilities: A1 to A2 adds direct editing and on-demand inspection, while A2 to A3 adds execution of the edited candidate before it is committed. Heilbronn Convex shows that the A1-to-A2 improvement is not explained by broader exploration: both arms attempt a similar range of strategies. The difference is how those ideas are incorporated. A1 frequently replaces large portions of the solver; in one matched trial, roughly one quarter of its scored edits regress and another third have no effect. A2 generally preserves the existing solver and adds components around it, reducing regressions to about one in eight, although it learns whether an edit worked only after evolutionary evaluation. A3 can test proposed changes through scratch exe- cution and regresses on roughly 4% of scored edits across the trials. Agentic editing therefore helps preserve prior functionality, while self-validation turns more of the budget into measured refinement. DABStep reveals why the relative gains differ across tasks. A1 sometimes identifies real instruction problems from injected context, but cannot inspect the database or run the revised agent. A2 can query the database, allowing it to correct fee-rule enumeration; the same accepted edit also tight- ens the response contract for exact-match answers. Because both changes affect many questions, they raise train/test accuracy from 33.7%/31.3% to 67.3%/65.3%. This closes roughly 55% of the A1-to-A3 gap, compared with about 22% on Heilbronn Convex. A3 additionally deploys and tests the edited agent, allowing it to validate a structural replacement for the fee-computation procedure and reach 95.3%/91.3%. Thus, database-grounded editing can capture broad instruction-level im- provements, whereas executable-code and dependent-object rewrites benefit most from end-to-end validation. D.3HISTORY-ACCESS INTERFACE We test two ways to give the mutator access to earlier search: a short ledger injected into each turn and structured tools for retrieving prior programs and traces. The four arms compare each mechanism alone and both together on Signal Processing and DABStep. Setup (Signal Processing). The selected parent’s code and evaluation are available in every arm; this ablation changes access to other programs’ history. B1 (parent-only) receives no additional history. B2 (ledger-only) receives recent approaches, score changes, stagnation information, and the active paradigm in its prompt, but cannot fetch prior programs or traces. B3 (tools-only) can request earlier programs, lineage, scores, and thinking traces through evo.py, but receives no ledger. B4 (full) has both mechanisms. All four arms use Sonnet-5 for 100 iterations with the same population, islands, evaluator, and model-facing mutation budget. The evolution graph and reflector/proposer are off throughout. B4 therefore uses the published Signal Processing history-access interface without the reflector/proposer. Results (Signal Processing). Table 11 reports the best overall score reached in one trial per arm. ArmInjected ledgerStructured history tools overallscore B1 parent-onlyoffoff0.6614 B2 ledger-onlyonoff0.7074 B3 tools-onlyoffon0.7487 B4 fullonon0.7681 Table 11: History-access interface ablation, Signal Processing. One 100-iteration Sonnet-5 trial per arm, reflector/proposer off throughout. Best overallscore reached within the 100-iteration budget (iterations 73, 97, 99, and 75 respectively). Analysis (Signal Processing). Scores rise from 0.6614 for B1 to 0.7074 with the ledger alone (B2) and 0.7487 with tools alone (B3). B4, with both mechanisms, reaches 0.7681. Thus each 20 mechanism helps relative to parent-only in these runs, and the combination scores highest. The B4 score is near the published Sonnet-5 mean of 0.7656 (best 0.7841), although the reflector/proposer is disabled in this ablation. The four final Signal Processing programs use different filtering designs. B1 combines a quadratic ridge-trend filter with a noise-adaptive moving average. B2’s champion describes an adaptive Kalman filter. B3 adds Rauch–Tung–Striebel backward smoothing to a Kalman filter. B4 combines Kalman and Savitzky–Golay smoothing with a median prefilter and slope-hysteresis logic. The inspected winning turns illustrate how the interfaces are used. B1, with no access to other attempts, proposes a larger-window Savitzky–Golay filter and Holt smoothing, tests both, rejects them, and keeps a small parameter change. B3 first queries the best programs, recent scores, and lineage, then verifies a refinement of its Kalman/smoothing filter with local score sweeps. B4 rec- ognizes a plateau of more than thirty iterations, queries high-scoring and diverse programs, and uses a local parameter sweep to find an improvement at a narrow threshold boundary. The B4 turn shows the ledger’s broad progress summary and targeted history lookup being used in the same edit. Setup (DABStep). DABStep uses the same four ledger/history-tool settings as Signal Process- ing. Its evolution graph and reflector/proposer remain enabled in every arm. The graph selects target questions on the controller side, while this ablation changes the history visible to the mu- tator; turning the graph off would change a second part of the loop. The reflector/proposer also stays fixed across arms. For B1 and B2, the core evo.py history commands and the task- specific get performance, getsuggestedquestions, and getartifacts history commands are removed. B3 and B4 retain them. All four arms keep the six commands used to in- spect and test the current candidate: query, test agent, analystquery, checkdeploy, executesql, and testdraft. Results (DABStep). We report one primary 100-iteration Sonnet-5 trial per arm. Each champion is freshly scored on all 300 training and 150 held-out test questions. B4 reuses the published flagship champion. A second B4 run is discussed below. ArmInjected ledgerStructured history toolsTrain (of 300)Test (of 150) B1 parent-onlyoffoff234 (0.780)116 (0.773) B2 ledger-onlyonoff210 (0.700)99 (0.660) B3 tools-onlyoffon242 (0.807)119 (0.793) B4 fullonon286 (0.953) 137 (0.913) Table 12: History-access interface ablation on DABStep. One primary 100-iteration Sonnet-5 trial per arm; each champion is freshly scored on the full 300-question training and 150-question test pools. Rows follow B1–B4 arm order. Analysis (DABStep). The four primary runs show a different pattern from Signal Processing. Ledger-only B2 scores below parent-only B1 on both train (210 versus 234) and test (99 versus 116). Tools-only B3 is only slightly above B1 (242 train and 119 test). Full B4 reaches 286 on train and 137 on test, the highest result in this comparison. On DABStep, the large observed gain appears when both history-access mechanisms are enabled; neither mechanism alone shows a comparable gain. This is the central result of the DABStep four-arm table. A useful fix lost at the evaluation gate. The B1 and B2 traces show that discovering a useful fix does not guarantee it survives search. Both arms repeatedly identify a seed instruction that lists only five merchants from an older dataset version. For an unlisted merchant, the seed can answer “Not Applicable” without calling a tool, even though the current data contains thirty valid merchants. B1 repairs the instruction, and its assigned target question also improves, so the edit enters the surviving lineage. B2 makes and tests the same kind of repair in its first iteration, but its assigned target is unrelated to the merchant bug; the evaluation gate rejects the edit, and later candidates still inherit the stale instruction. 21 The ledger helps when the mutator can inspect prior programs. The ledger gives the mutator a live-updated record of earlier approaches and projected scores. Its score column is revised as the similarity mechanism estimates full-pool performance, so a ledger value can differ from the program’s immediate measured result. In one traced case, a program scored 0.1767 on its immediate evaluation but later appeared in the ledger at 0.4011. B1 sees no such history table. B2 receives the changing projections on every turn but cannot retrieve the earlier programs and traces needed to judge what a high-scoring entry actually does. The ledger-only arm is therefore exposed to uncertain historical scores without a way to check the underlying work, which helps explain its weaker result. B3 and B4 can inspect that work through their history tools. How B3 and B4 use history. B3 trails B1 through much of the search and gains most in its final thirty iterations. Before two late improvements, the mutator queries prior programs, lineage, scores, and thinking traces. It then builds a counterfactual simulator for questions about alternate fee schemes and lookup cubes for card-scheme and ACI questions, testing both before acceptance. B1 also attempts substantial late changes, including a case bank for difficult questions; the distinction in these traces is that B3 consults the population’s history before deciding where to work, not that B1 is incapable of complex edits. B4 reaches B1’s final score by roughly iteration 40 and improves further. Around iteration 36, it queries earlier high-scoring programs and suggested questions, then builds and checks a vectorized aggregation engine for group-by questions. The checks expose an unsupported array/object param- eter type, which the mutator works around before the edit is kept. Around iteration 84, it adds a computation mode for ACI-selection questions and checks it against known cases. These episodes show the full arm using its history tools while carrying out substantial, verified edits. Variation within the full configuration. A second B4 run uses the same configuration and model as the flagship but scores 248/300 on train and 114/150 on test, rather than 286/300 and 137/150. It ends near B3’s primary 242/300 and 119/150 result despite having both history-access mechanisms available. The two B4 runs have similar Bash-call volume per iteration (32.3 versus 33.7), but allocate those calls differently. In the flagship, 2092 of 3229 Bash calls invoke evo.py and 974 invoke evo sweep.py, about 95% combined. In the second run, 1671 of 3367 invoke evo.py and 127 invoke evosweep.py, about 53% combined. The remaining calls, many of which inspect the edited blueprint directly, are far more common in the second run (about 47% versus 5%). The tool-use pattern and outcome vary together in these two runs; tool use was observed rather than assigned as another ablation treatment. D.4SIMILARITY-MANIFOLD TARGETING We test whether the true question-similarity graph improves DABStep target selection and expan- sion. The full semantic configuration (C3) scores well above the uniform and shuffled controls in these runs. Setup. All three arms use DABStep’s 300-question training pool and keep the mutator, history- access interface, parent policy, first-evaluation gate, regression-sample size, expansion-question count, scout count, and acceleration budget fixed. The evolution graph remains enabled in every arm. The arms differ in how they select a target and expand evaluation to previously untested ques- tions, as described below. C1 (uniform matched control) samples the target uniformly from the same unsolved candidate set, keeps the regression step identical, and evaluates the same number of random previously untested questions, including the same scout count; persistent solved-state, donor registry, and target-attempt counts are retained. C2 (shuffled similarity graph) runs C3’s identical information-gain formula, but reads the similarity term off a symmetric index permutation of the real matrix rather than the real values — same value distribution, wrong question pairing, isolating whether the specific semantic geometry matters rather than merely having some structured (non-uniform) sampling bias. C3 (full semantic manifold) selects the target by information gain over questions the parent has not solved, runs the usual regression sample from the parent’s cumulative solved set, and expands the search to semantically similar previously untested questions; archive-wide solvers may be retrieved as donors. All three arms share the same regression-risk sampler: whatever target a given arm’s rule selects, 22 that target’s most at-risk previously-solved questions are drawn from the true, unpermuted similarity graph. Only target selection and untested-question expansion differ across arms. C3’s champion is the flagship run already reported in Table 1; we report a fresh full-pool re- evaluation of that same champion here, rather than the in-run bookkeeping trajectory used elsewhere, to match C1/C2’s methodology exactly. Results. One 100-iteration Sonnet-5 trial per fresh arm (C1, C2); C3 and A1 reuse the flagship and seed champions already reported elsewhere, each freshly re-evaluated against the same full pools as C1/C2. ArmTrain (of 300)∆Test (of 150)∆ A1 seed (no evolution)101–47– C1 uniform (matched)95−642 −5 C2 shuffled similarity113+1251+4 C3 information-gain (flagship)286+185137+90 Table 13: Similarity-manifold targeting on DABStep. C1 and C2 each use one 100-iteration Sonnet- 5 search; C3 reuses the published flagship and A1 the seed. All four champions are freshly scored on the full 300-question training and 150-question test pools. ∆ is relative to A1. The flagship’s in-run trace reached 293 on train, while its independent full-pool re-score here is 286. Analysis. C3 reaches 286/300 on train and 137/150 on test, well above C1 (95/300, 42/150) and C2 (113/300, 51/150). C1 scores below the seed (101/300, 47/150), while C2 is only slightly above it. A repeat held-out evaluation of C2’s champion scored 49/150 rather than 51/150, under- scoring the small size of that margin. The per-question comparison in Table 14 shows both more gains and fewer regressions for C3: it gains 189 questions and regresses 4, versus 29 gained/35 regressed for C1 and 28 gained/16 regressed for C2. Most gains in every arm transfer beyond its directly targeted questions (C3: 143/189; C1: 25/29; C2: 22/28). All arms use the true graph for regression-risk sampling, so aligning target selection and expansion with that graph may help protect previously solved neigh- bors. These runs evaluate those two semantic operations together; they do not separate their contri- butions. ArmGainedRegressedNetGain via targetGain via transfer C1 uniform (matched)2935 −6425 C2 shuffled similarity2816+12622 C3 information-gain (flagship)1894+18546143 Table 14: Per-question diff against the A1 seed baseline on the training pool (300 questions), by arm. Net is Gained minus Regressed. Gain via target/transfer partitions each arm’s Gained count by whether that question was ever one of the arm’s own Phase-0 targets across its 100 iterations. EPROMPTS USED CoCoEvolve uses Cortex Code as its mutator, alongside a reflector and proposer. The mutator edits the candidate in its workspace. The reflector chooses keep, new, or none from the approach ledger; new invokes the proposer to name a strategy. The excerpts below show the instructions relevant to these roles. The math, ADRS, and Signal Processing mutators share the same generic interface: an injected ledger, on-demand evo.py history queries, and a task workspace for editing and execution. DAB- Step replaces the generic system message with a Cortex-agent builder prompt that adds warehouse tools. The data engineering prompt defines the scaffolding of the context-curation pipeline that CoCoEvolve improves. 23 Mutator user (math / ADRS) Richer historical evidence is available on demand through a CLI: python3 evo.py get_best_programs -n 5 python3 evo.py get_lineage --id <pid> python3 evo.py get_program --id <pid> python3 evo.py get_recent_scores -n 20 The current program is already written to a file in this directory. Edit that file directly. Do not paste code into the reply. Before finishing, write the declared approach to APPROACH.txt: the single word hillclimb or new. Mutator system (DABStep workspace) You are an expert Snowflake Cortex Agent builder. You evolve agent blueprints to improve accuracy on payment processing questions. Diagnosis priority: test before you diagnose. Run the existing tool the failing question would use (query, test_agent, or analyst_query). If it returns the right result, the fix is routing; if not, the fix is the tool itself. Live investigation tools (python3 evo.py <subcommand>): query -- read-only SQL against live tables test_agent -- run a scratch clone of the current agent check_deploy -- compile-check a candidate DDL execute_sql -- CREATE/CALL against the warehouse test_draft -- deploy your edited program.json on question Q Mutator system (Data Engineering) You evolve the ** KG construction pipeline ** for data-eng-bench tasks. ## What a Knowledge Graph is (and why it matters) Each data-eng-bench task is a small analytics engineering challenge: given a database and an ‘instruction.md‘, an agent must write dbt models that pass pytest. Before that coding step runs, ** step 1 ** produces a ** Knowledge Graph ** | a structured markdown file (‘KG.md‘, ≤300 lines) that acts as a briefing document for the coder. A KG that only restates ‘instruction.md‘ is useless. A good KG is warehouse-grounded and captures: - What the task asks for (outputs, metrics, grain) - Which schemas/tables/columns exist and how they join - Model layering (staging → intermediate → mart), macros, and business rules - Edge cases, validation checks, and pitfalls to avoid A separate ** Harbor coding agent ** (fixed, not under your control) reads ‘KG.md‘ and writes dbt SQL. Before scoring, it may iterate against ** KG-derived self-tests ** (‘test_kg.py‘ compiled from that KG after step 1). 24 Those self-tests are NOT the grader | the hidden verifier is unchanged. ** Your ** job is to design ** one ** KG factory reused on every data- eng-bench task | the pipeline that * creates * that KG (and the lean ‘kg-adherence-test-author‘ protocol that shapes how tests are derived) via skills and agent orchestration descriptions. Better architecture → richer KGs + granular self-tests → higher pytest pass rate. ## The pipeline explained Your pipeline ** explores the task environment and produces ‘KG.md‘, which is handed off to a different, fixed agent that solves the task. ** That is the only fixed contract. Everything else | how many agents do the exploring/writing, and ** what reusable skills the KG factory follows | is yours to design. ** Harbor step 2 receives ** only ** ‘ KG.md‘, not catalog or runtime skill files. The skill catalog ** SHIPS with three section-skills ** (‘kg-agent-usage-guide‘, ‘kg-metrics-and-business -rules‘, ‘kg-structure‘). A fourth catalog key, ** kg-adherence-test-author ** , is compile-time only (lean protocol for deriving granular test_kg.py from each KG | not a KG.md section). Your job is to ** EDIT or GROW the existing pipeline ** : author NEW skills and DIVIDE THE LABOR across agents or REFINE/IMPROVE the existing skills. Refine kg-adherence-test-author when traces show tests are too generic or Harbor ignores the self-test loop. Drop any skill that stops earning its place. ## Two axes of architecture improvement ** Axis 1 | Agent topology ** (who runs where, doing what) - Edit ‘agents[]‘: an ordered list of sequential agent sessions. - The seed is ONE ‘architect‘ that explores and writes ‘KG.md‘ in the same session (query results stay in context). This is a STARTING point. Do not strip the warehouse look | a factory that writes from the instruction alone produces a useless briefing. - Add, split, merge, or reorder agents freely. Split into explorer → architect only if ‘explore_coverage‘ shows the look was skipped and a separate look step would help | a handoff file loses raw tool output. - Write each agent’s ‘prompt‘ (explore the DB, synthesize findings , verify facts, render the final KG, etc.). - Attach proven skills to specific pipeline steps via ‘agents[]. skills‘. Skills guide KG exploration and authoring inside the factory; Harbor step 2 receives ** only ** the generated KG (not catalog or runtime skill files). 25 Agents share one workspace. Prefer look+write in the same session so ‘sql_execute‘ results stay in context | a handoff file loses raw tool output. ** Axis 2 | Skills ** (what reusable protocols the team needs) - Add, edit, or remove keys in ‘skills‘: ‘description‘. Attach each skill to a pipeline step via ‘agents[].skills‘. - The assigned agent writes the full skill this eval → ‘runtime_skills/<name>.md‘. You never paste markdown into the spec . ## Evidence (how to improve either axis) Do not guess topology or skills. Justify an Axis 1 or Axis 2 edit with a recurring gap in the parent eval: Harbor traces (‘agent_trace.txt‘), verifier stdout (‘verifier_output_raw.txt‘), and/or ‘explore_coverage‘. ## Full system (three platform steps) Step 1 | KG pipeline (you design) → ‘KG.md‘ Step 1b | Platform compiler (fixed) → ‘test_kg.py‘ from KG + ‘kg- adherence-test-author‘ Step 2 | Harbor agent (fixed) → dbt models + self-test iteration → pytest score ## Your artifact: ‘_PIPELINE_SPEC‘ Edit between ‘# EVOLVE-BLOCK-START‘ and ‘# EVOLVE-BLOCK-END‘: agents | ordered steps: name, prompt, skills[] skills | catalog of description per key (justification optional; assigned agent writes .md) ## Constraints - Valid Python dict; generic templates only (no task-specific table/column names). - Agent prompts keep ‘task_name‘, ‘db_name‘, ‘timeout_sec‘ placeholders. - Final KG ≤300 lines; no solution SQL in the KG. - Never put markdown skill bodies in ‘skills‘ | metadata only. Reflector system You are a strategy-pool controller for an evolutionary code-optimization run. One worker slot has just freed up and you decide ONE thing for it: feed it into an existing active strategy, spawn a brand-new strategy (a proposer agent will design it), or run plain hillclimbing with no named strategy. You do NOT design the new strategy yourself and you do NOT edit code. Proposer system You are an expert algorithm researcher and strategic advisor. Suggest breakthrough algorithmic IDEAS in a JSON object with an ’ideas’ array. Each idea describes a high-level strategy (not code). Focus on ideas that are fundamentally different from what has been 26 tried. You MUST always return at least one idea. FAGENT OPTIMIZATION F.1DETAILED METHODOLOGY DABStep optimization uses the InfoGain branch of Algorithm 1 over a 300-question training pool, with a disjoint held-out test split reported in Table 1. The run behind Table 15 uses a 100-iteration budget, one island, population 40, cascade threshold [0.005], a task workspace, evo.py lineage tools, and the warehouse overlay of Appendix C.2. The seed p 0 is a Cortex Agent blueprint with instructions, payment analytics text-to-SQL, docssearch, and a single-transaction fee calculator. The mutator may rewrite the instructions, tool schemas, stored procedures, and dynamic tables. GEPA receives the same specification but optimizes only its instruction text. A post-hoc full-pool evaluation of its champion scores 142/300 on train and 59/150 on test (Table 1). Meta-Harness (Lee et al., 2026) receives the same seed and 100-iteration budget through a faithful port of the Terminal-Bench-2 loop. The proposer prompt contains only a file pointer and receives no curated history or pre-injected per-question signal, matching the original design. We make two changes to the port. Cortex replaces the Anthropic API as the proposer backend, and the pooled- parallel DABStep evaluator replaces Harbor sandboxes. Each child is a Cortex Agent JSON speci- fication scored on all 300 training questions. The main-paper Meta-Harness row uses the self-test variant. Its proposer receives testdraft, testagent, and checkdeploy, which provide the same self-validation surface as CoCoE- volve’s A3 arm (Appendix D.2). A held-out re-evaluation of the champion produces 80.0% easy accuracy (20/25), 78.4% hard accuracy (98/125), and 78.7% total accuracy (118/150), with no evaluator errors. These are the values in Table 1. Figure 3 plots best-so-far train accuracy against iteration for all three methods. The remaining two rows are no-search baselines, each scored once on all 450 questions with the same evaluator and scorer as the search methods. The Snowflake Cortex Code Agent row scores the unmodified seed; the Stock Cortex Agent row scores a general-purpose coding agent without the seed’s specification. Snowflake Cortex Code Agent redeploys the unmodified seed p 0 .Its instructions, payment analytics text-to-SQL tool, docssearch, and feecalculator remain un- changed. This row measures what the seed already solves before optimization. Stock Cortex Agent removes the Cortex Agent product entirely. A single Cortex Code CLI call using claude-sonnet-5 answers each question in a fresh session without memory or an evo- lutionary loop. The model receives the DABStep documentation and read-only SQL access to the live tables. It does not receive a semantic view, text-to-SQL tool, fee-calculation procedure, or the seed’s domain-specific instructions. Together, these two rows bound performance without search: the first retains the seed specification, while the second retains only a general-purpose coding agent with raw data access. F.2COMBINED-SUITE LEADERBOARD Table 15 places our combined-pool evaluations beside public DABStep leaderboard entries (Egg et al., 2025) and the context-engineered configuration of D ̈ ohmen et al. (2026). Our CoCoEvolve, Meta-Harness (Lee et al., 2026), GEPA (Agrawal et al., 2026b), and no-search rows are evaluated on the 300 training and 150 test questions together; the Meta-Harness and GEPA rows use the champions from Table 1. Public leaderboard submissions report combined-suite accuracy without a separate train/test breakdown. The table is ranked by hard-question accuracy. F.3WHY SELF-TEST TOOLS DID NOT PREVENT META-HARNESS’S CHURN Figure 3 shows Meta-Harness’s self-test variant landing its one real gain at iteration 53 and then failing to improve for the remaining 47 iterations despite holding the same 27 MethodEasy (%)Hard (%) CoCoEvolve87.5095.24 MotherDuck, full-context + data modeling (D ̈ ohmen et al., 2026)93.1093.20 NVIDIA KGMON (NeMo Agent Toolkit) Data Explorer87.5089.95 DataPilot86.1187.57 Meta-Harness84.7277.25 g-agent-gpt5-1104-188.8962.96 CambioML energent.ai DS Agent94.4457.67 GEPA88.8936.24 Snowflake Cortex Code Agent75.0024.87 Stock Cortex Agent (baseline)56.9411.90 Table 15: Accuracy on the combined 450-question DABStep suite, ranked by hard-split accuracy. 0255075100 Iteration 0.0 0.2 0.4 0.6 0.8 1.0 Best train accuracy (/300) ...then churns on repeated deploy/eval failures, no further gain Meta-Harness self-diagnoses a deploy/eval bug here CoCoEvolve 0.977 Meta-Harness (self-test) 0.783 GEPA, final only 0.473 CambioML, external/combined-450 0.636 Figure 3: DABStep best-so-far train accuracy (/300) versus iteration. CoCoEvolve and Meta- Harness (self-test) are running-best curves from their evolution logs. GEPA has no per-iteration full-pool trace because its search evaluates small subsamples, sometimes only one question per step, so its final post-hoc score appears as a dashed reference line. The second dashed line shows Cam- bioML energent.ai DS Agent, an external combined-pool leaderboard submission without a matched budget or train-300 trajectory. Its easy and hard accuracies are 94.44% and 57.67%, respectively (Table 15). Appendix F.3 analyzes Meta-Harness’s post-iteration-53 stall. testdraft/testagent/checkdeploy self-validation surface throughout. We inspected the archived per-iteration tool-call transcripts to understand the stall. They show two recurring problems: repeated searches for nonexistent question tables and semantic-view edits that pass the preliminary check but fail at deployment. These traces explain failures in this run; they do not establish that either problem would persist under every use of the self-test tools. The query tool works; what recurs is a dead end nobody flagged, relearned every time. Meta- Harness can query the real warehouse schema successfully. At iteration 61, its proposer runs DESCRIBE TABLE DABSTEP PAYMENTS and receives a valid column list. In the same itera- tion, it reads DABSTEPFEES and DABSTEPTXNFEEFACT correctly. The recurring failure is different. On each occurrence, the proposer spends nine calls looking for DABSTEP QUESTIONS, DABSTEPEVALQUESTIONS, and DABSTEPEVAL. These tables do not exist because the question and answer bank is stored only in frontierval.json and evolutionsummary.jsonl. The prompt tells the proposer to read these files for per-question pass rates, but it never states that they are the only source of question-level information. 28 Both harnesses start from the same initialprogram.json, agentspec,and referenced tables.The difference is therefore not the tool or seed.It is what each prompt surfaces on every turn and whether a discovered fact persists. The self-test tool cannot see the specific bug class that recurs. The DDL failure is structural. A Cortex Analyst semantic view contains two representations that must remain synchronized: an outer SQL tables()/relationships() clause and an embedded Cortex Analyst extension JSON object. The JSON separately lists tables, facts, and dimensions. At iteration 61, the proposer edits both representations with ad hoc Python string operations. It inserts a relationship into the outer clause and separately patches the JSON table list instead of generating both from one source of truth. The two representations can therefore drift apart. In this case, a table appears in tables() but not in the JSON. The resulting text is syntactically well formed, so check deploy cannot reject it in advance. That tool checks whether the DDL compiles but does not perform a data-flow check (mh tools.py). It can catch syntax failures, as it did for an unescaped quote in an earlier iteration, but it cannot validate cross-references between the two representations. Only the live Snowflake compiler exposes that error, and it returns only a line and position rather than a diff. The observed failures point to distinct remedies. A prompt that identifies where question-level data lives and carries learned dead ends forward could reduce repeated fruitless queries. Structural valida- tion of the semantic view’s two representations could catch the DDL mismatch before deployment. In this run, self-test access alone did not supply either safeguard. F.4EXAMPLES OF BEST PROGRAMS DABStep’s artifact is a Cortex Agent spec (Snowflake Inc., 2026a). The champion reached an in-run train estimate of 293/300 (combined score 0.977). A fresh full-pool re-evaluation of the same artifact scored 286/300, as reported in Table 13. While the mutator discovered a suite of optimizations across the specification (including refined verified queries, precise orchestration instructions for disambiguation, and injected domain context), we highlight the architectural shift we believe contributed most to its performance: demoting the text-to-SQL surface in favor of a deterministic compute kernel. By routing fee, ranking, and glossary questions around the LLM entirely, it avoids freehand SQL errors. The champion agent achieved this by rewriting the custom tool from a 5KB SQL procedure to a 53KB Python engine with 14 modes, backed by NumPy, SciPy, and NetworkX. The agent can therefore answer complex questions with a single tool call instead of summing many single-row fees in context, directly addressing the failure mode discussed in Section 3. The deployed agent reads the following abridged specification, which heavily guides it to trust its new modes over its text-to-SQL capabilities. Key routing instructions and tool descriptions added by the mutator are highlighted in blue: Evolved specification (champion, abridged) "agent_spec": "instructions": "response": "Provide precise, data-driven answers. Always show the exact numbers from query results. <abridged...>", "orchestration": "You are a Payment Operations Analyst agent specialized in the Dabstep payment processing dataset. <abridged ...> ## Tool Selection Rules ### New fee_calculator modes for residual question families Route these DIRECTLY to fee_calculator instead of freehand SQL via payment_analytics: - "For [account type X and/or MCC description: Y...] what would be the average fee..." -> mode=’avg_fee_criteria’ 29 - "Which fee ID(s) apply to <criteria>" with no merchant scope -> mode =’fee_ids_criteria’ - "How many unique/distinct <column> are there" -> mode=’ distinct_count’ <abridged...> ## Fee Rule Engine (fee_calculator tool) -- generalized modes The fee_calculator tool is a generalized vectorized fee-matching engine over a merchant’s real transactions (it sums ALL matching fee rules per transaction, then aggregates). Route these question categories to it directly instead of hand-writing SQL: <abridged ...> Trust this tool’s numbers over manually derived SQL for these categories -- it already handles monthly fraud/volume bracket computation, capture_delay bucketing, and the correct intracountry definition. <abridged...>" , "tools": [ "tool_spec": "type": "cortex_analyst_text_to_sql", "name": "payment_analytics", "description": "Use this tool to query payment transaction data, merchant information, fee rules, and acquirer details. < abridged...>" , "tool_spec": "type": "cortex_search", "name": "docs_search", "description": "Use this tool to search payment processing domain documentation including the merchant guide, fee optimization strategies, fraud prevention best practices. < abridged...>" , "tool_spec": "type": "generic", "name": "fee_calculator", "description": "Generalized vectorized fee-rule engine for a merchant’s transactions. Use this tool (NOT freehand SQL) for ANY question about: (1) which card scheme a merchant should steer traffic to for minimum/maximum fees -- mode=’steer’; (2) total fees a merchant paid/should pay over a year/month/day -- mode=’ total’; <abridged...>", "input_schema": "type": "object", "properties": "p_mode": "description": "One of: ’steer’ (best/worst card scheme for min/max fees), ’steer_aci’ (best/worst ACI to move a merchant’s fraudulent transactions towards for min/max fees... < abridged...>", "type": "string" <abridged...> ] 30 A warehouse stored procedure backs the feecalculator tool. While we omit the full 53KB Python body, the evolution trajectory reveals several striking mechanistic and semantic discoveries that the agent authored and verified via workspace execution: • Vectorized tensor contraction. The initial tool used SQL to evaluate 9-field conditional fee rules. The champion replaces this with a sparse boolean tensor contraction in Python. It precomputes a boolean “allowed” matrix for each field where wildcards become an all-True row, combines them via bitwise &, and computes the final fees using sparse matrix-vector products (scipy.sparse.csr matrix). • Semantic bug fixes. The mutator fixed two silent bugs in the human-authored seed. First, it corrected a whitespace bug in a string label (’> 5’ to ’>5’) that had silently disabled 92 fee rules for 6 merchants.Second, it redefined “intracountry” transactions from IP COUNTRY = ISSUINGCOUNTRY to ISSUINGCOUNTRY = ACQUIRERCOUNTRY, a definitional shift that alters the evaluation of 70% of the 138,236 rows in the dataset. • Emergent domain semantics. The agent discovered through trial and error that fees stack. Its code comments explicitly warn its future self: “because fees STACK... Do NOT take the MAX of a single matching rule... that produces a different (and wrong) ranking.” • Graph-based join resolution.Rather than letting the orchestrator guess join keys for text-to-SQL, the tool hardcodes the schema’s foreign-key graph and uses networkx.shortest path to emit deterministic JOIN ... ON fragments, return- ing an explicit refusal if no path exists. • Regex over semantic search. For exact-definition questions, the agent deliberately re- places vector search with a regex and difflib parser over the documentation, reasoning that embeddings might return a topically-related chunk that omits the exact definition. Beyond the champion tool shown above, other evolutionary lineages explored alternative structural solutions. The most notable alternatives materialized dynamic tables to pre-compute frequently- joined columns and eliminate flaky on-the-fly joins and produced detailed sets of verified queries encoding domain-specific reasoning paths. While these alternative approaches were interesting, they ultimately lost to the champion’s centralized Python compute kernel. Across all lineages, the agent also continually refined its orchestration guidelines, encoding lessons from observed failures into tight situational guardrails. GAI FUNCTIONS G.1SYNTHETIC DIAGNOSTIC: LOCKPICKING WITH INTERFERENCE Construction and protocol. This diagnostic isolates recovery of interacting rules from the seman- tic uncertainty of the natural-data tasks. Each example is a six-digit string P 1 ...P 6 with an integer label in 0,..., 6. A position first passes a position-specific candidate-set check; twelve ordered trigger rules (24 trigger–target edges) can then invalidate otherwise-valid positions. The label is the number of positions that remain valid. For example, P 1 ∈1, 4, 7. Two rules that can subsequently invalidate P 1 are P 2 = 8 =⇒ P 1 ,P 3 invalid, P 3 = 3 =⇒ P 1 ,P 2 invalid. The seed provides all six candidate sets but only one of the twelve interference rules. Every method receives the same 50 labeled strings and 50 mutation iterations. There is no held-out partition: train and test are the same fixed set, because the measured quantity is exact reconstruction of the latent rule system rather than distributional generalization. 31 Why aggregate accuracy is underdetermined. The evaluator returns only the final count, not which positions were invalidated. In the example above, if both triggers fire, omitting either edge into P 1 need not change that row’s label. A trigger can likewise be masked when its target was already rejected by the candidate-set check or an earlier rule. Consequently, many structurally dif- ferent rule tables agree on most of the 50 examples. This property makes the diagnostic useful: a locally successful patch can raise accuracy without identifying the rule responsible for the improve- ment. MethodGEPAOpenEvolveAdaEvolveCoCoEvolve Correct / 5041274550 Accuracy0.820.540.901.00 Table 16: Accuracy on Lockpicking with Interference after 50 iterations. Recovered search behavior. CoCoEvolve reaches 50/50 after six mutations by contrasting mul- tiple failures and compiling all twelve recovered rules into deterministic SQL. OpenEvolve ends at 27/50 with unresolved contradictions in its own worked examples. AdaEvolve reaches 45/50 at it- eration 34 and then plateaus: only two of its twelve learned trigger rows have the correct target pair, five reuse the same [P3,P5] shortcut, and none can invalidate P 1 . The gap is therefore structural rather than evaluation noise: the baselines fit surrogate rules that score well on the fixed set, whereas CoCoEvolve recovers the latent rule system. G.2DETAILED METHODOLOGY Program representation. Each candidate is an executable Snowflake SQL expression that forms the body of a scalar AI function. The seed for every task contains one AICOMPLETE call, a task instruction, a row-dependent user message, and a structured-output schema. Evolution may rewrite the instruction and schema descriptions, introduce CTEs or deterministic SQL post-processing, and compose multiple model calls, but it must preserve the function signature, output field and type, and final accessor. Thus the search space includes both prompt optimization and pipeline optimization while every candidate remains directly executable by the target query engine. The seed is deliber- ately minimal: for example, the go emotions seed asks for every emotion that “clearly applies,” and the multilexnorm seed gives only four normalization examples. Tasks and metrics. The five live-data tasks use the following row-level objectives: goemotions uses example-level multi-label F1; finqa and convfinqa use execution accu- racy after canonicalizing the returned numeric answer; lexglue uses exact match over thirteen Supreme Court issue areas; and multilexnorm uses exact string match, including case, punc- tuation, and spacing. The synthetic lockpicking 6 diagnostic uses exact match on the integer score. A malformed SQL body, invalid structured output, or warehouse execution failure receives score zero together with the compilation or execution diagnostic. Data partitions. For each live-data task, the search pool is partitioned once with seed 42. Two thirds form a validation split used only for fitness and acceptance, while the remaining third supplies row contents and scorer feedback to the mutation agent. The disjoint test table is never exposed during evolution; after the 50-iteration run, the exported validation champion is re-executed on that test table to obtain Table 2. This separation is important because the mutator can inspect concrete training failures. lockpicking6 is intentionally different: all examples are visible and scored because its purpose is to test whether search can reconstruct a hidden rule system rather than gener- alize to an unseen natural-data split. Mutation protocol. The AI-function runs instantiate the general loop in Appendix C.1, the history interface in Appendix C.2, and the ℓ p roster in Appendix C.5. Their task-specific specialization is a fresh-session curator–mutator split: the curator inspects the roster, ledger, and training-only failure history to select a parent, an optional merge partner, and up to ten representative rows; the mutator receives only that bounded packet and may verify its SQL on those rows up to three times. The subsequent full-validation evaluation is the sole admission gate. Runs use 50 mutation attempts, one 32 island, population 40, archive capacity 20, and eight parallel workers; similarity-informed targeting is disabled because every child is evaluated on the complete validation split. Population objective. AI functions use the continuous p = 2 instantiation of Appendix C.5, with a roster budget of ten. Marginals are sums of row-level score improvements rather than binary solved-row counts; all other roster construction, consolidation, and parent-selection details follow that section. Matched baselines and score provenance. OpenEvolve and AdaEvolve start from the same task- specific SQL body and receive 50 mutation attempts. Their validation champions are scored post hoc on the same held-out test partition. GEPA likewise optimizes the SQL body against the validation partition; Table 2 uses the best completed budget for each task. For the five live-data tasks, the paper values are held-out re-evaluations rather than the optimizer’s internal archive fitness. For lockpicking, all methods are scored on the same fixed visible set, consistent with the diagnostic protocol above. G.3EXAMPLES OF BEST PROGRAMS The evolved programs remain ordinary SQL bodies, but they are substantially more operational than their seeds. Across tasks, evolution turns a generic instruction into an explicit decision procedure: ordered rules for classification, reference and sign conventions for numerical reasoning, or token- preservation contracts for normalization. We give representative examples below, reproducing the two lockpicking champions in full and abridging the longer live-task programs. For the live tasks, the base boxes show the task-bearing seed instruction; we omit repeated label enumerations, the AI COMPLETE wrapper, and the unchanged structured-output schema. lockpicking6: exact rules versus a surrogate ruleset. The two champion artifacts make the structural difference directly inspectable. The CoCoEvolve listing is the complete deterministic scoring expression produced by its winning child. In the deployed AI-function body, the resulting integer is passed to AICOMPLETE only for schema-conforming serialization. lockpicking6 base program (instruction) Score a 6-digit program string. If the input is not exactly six digits, return 0. Valid digits: P1: 1,4,7 P2: 2,6,8 P3: 0,3,9 P4: 1,4,7 P5: 0,5,8 P6: 2,5,9 Apply interference rules in order: 1. IF P1 = 4 -> invalidate P2, P6 Return the number of positions still valid after both checks. CoCoEvolve champion: deterministic scoring expression CASE WHEN NOT REGEXP_LIKE(PROGRAM, ’ˆ[0-9]6$’) THEN ’0’ ELSE CAST( IFF(SUBSTR(PROGRAM,1,1) IN (’1’,’4’,’7’) AND NOT (SUBSTR(PROGRAM,2,1)=’8’ OR SUBSTR(PROGRAM,3,1)=’3’ OR SUBSTR(PROGRAM,5,1)=’5’ OR SUBSTR(PROGRAM,6,1)=’2’), 1, 0) + IFF(SUBSTR(PROGRAM,2,1) IN (’2’,’6’,’8’) AND NOT (SUBSTR(PROGRAM,1,1)=’4’ OR SUBSTR(PROGRAM,3,1)=’3’ OR SUBSTR(PROGRAM,4,1)=’7’ OR SUBSTR(PROGRAM,5,1)=’8’), 1, 0) + IFF(SUBSTR(PROGRAM,3,1) IN (’0’,’3’,’9’) AND NOT (SUBSTR(PROGRAM,1,1)=’1’ OR SUBSTR(PROGRAM,2,1)=’8’ OR SUBSTR(PROGRAM,4,1)=’7’ OR SUBSTR(PROGRAM,6,1)=’2’), 1, 0) + IFF(SUBSTR(PROGRAM,4,1) IN (’1’,’4’,’7’) AND NOT (SUBSTR(PROGRAM,2,1)=’6’ OR SUBSTR(PROGRAM,3,1)=’0’ OR SUBSTR(PROGRAM,5,1)=’5’ OR SUBSTR(PROGRAM,6,1)=’9’), 1, 0) + IFF(SUBSTR(PROGRAM,5,1) IN (’0’,’5’,’8’) AND NOT (SUBSTR(PROGRAM,1,1)=’1’ OR SUBSTR(PROGRAM,2,1)=’6’ OR SUBSTR(PROGRAM,4,1)=’1’ OR SUBSTR(PROGRAM,6,1)=’9’), 1, 0) + IFF(SUBSTR(PROGRAM,6,1) IN (’2’,’5’,’9’) AND NOT (SUBSTR(PROGRAM,1,1)=’4’ OR SUBSTR(PROGRAM,3,1)=’0’ OR SUBSTR(PROGRAM,4,1)=’1’ OR SUBSTR(PROGRAM,5,1)=’8’), 1, 0) AS VARCHAR) END 33 The AdaEvolve listing is its complete 45/50 champion prompt, including the ten worked examples accumulated during search. We normalize typographic dashes and operators to ASCII for typeset- ting; the wording and rules are otherwise unchanged. AdaEvolve champion: evolved executor prompt You are a scoring evaluator for a 6-digit program string. Output ONLY a single integer (0-6). ## CRITICAL RULE: Always use ORIGINAL digit values when checking triggers, even if that position is base-invalid. --- ## PHASE 1 - Input Validation If the input is not exactly 6 digit characters (0-9), output 0 and stop. Label positions left-to-right: P1, P2, P3, P4, P5, P6. --- ## PHASE 2 - Base Validity Lookup Mark a position BASE-INVALID if its digit is NOT in the listed set. | Position | Valid digits | |----------|--------------| | P1 | 1, 4, 7 | | P2 | 2, 6, 8 | | P3 | 0, 3, 9 | | P4 | 1, 4, 7 | | P5 | 0, 5, 8 | | P6 | 2, 5, 9 | --- ## PHASE 3 - Trigger Table Check EVERY row below independently. If the condition matches the ORIGINAL digit value at that position, mark ALL listed positions as TRIGGER-INVALID. Check all 12 rows every time - do not skip any. | Row | Condition | Mark TRIGGER-INVALID | |-----|-----------|----------------------| | 1 | P1 = 4 | P2, P6 | | 2 | P1 = 1 | P3, P4 | | 3 | P4 = 1 | P3, P5 | | 4 | P4 = 7 | P3, P6 | | 5 | P2 = 6 | P3, P5 | | 6 | P2 = 8 | P3, P5 | | 7 | P3 = 0 | P5, P6 | | 8 | P3 = 3 | P5, P6 | | 9 | P5 = 5 | P2, P6 | | 10 | P5 = 8 | P2, P6 | | 11 | P6 = 2 | P3, P5 | | 12 | P6 = 9 | P3, P5 | --- ## PHASE 4 - Final Count A position is VALID if and only if: - It is NOT base-invalid, AND - It is NOT trigger-invalid Count VALID positions and output ONLY that integer (0-6). --- ## WORKED EXAMPLES ** Example 1: ’729405’ ** P1=7, P2=2, P3=9, P4=4, P5=0, P6=5 Base: all valid (7 in 1,4,7, 2 in 2,6,8, 9 in 0,3,9, 4 in 1,4,7, 0 in 0,5,8, 5 in 2,5,9) Trigger scan (all 12 rows): 34 - Row 1: P1=7 != 4 -> skip - Row 2: P1=7 != 1 -> skip - Row 3: P4=4 != 1 -> skip - Row 4: P4=4 != 7 -> skip - Row 5: P2=2 != 6 -> skip - Row 6: P2=2 != 8 -> skip - Row 7: P3=9 != 0 -> skip - Row 8: P3=9 != 3 -> skip - Row 9: P5=0 != 5 -> skip - Row 10: P5=0 != 8 -> skip - Row 11: P6=5 != 2 -> skip - Row 12: P6=5 != 9 -> skip No triggers fired. All 6 positions valid. ** Output: 6 ** --- ** Example 2: ’429405’ ** P1=4, P2=2, P3=9, P4=4, P5=0, P6=5 Base: all valid. Trigger scan: - Row 1: P1=4 -> FIRES -> mark P2, P6 TRIGGER-INVALID - All other rows: no match Trigger-invalid: P2, P6 P1=VALID, P2=INVALID, P3=VALID, P4=VALID, P5=VALID, P6=INVALID ** Output: 4 ** --- ** Example 3: ’720105’ ** P1=7, P2=2, P3=0, P4=1, P5=0, P6=5 Base: all valid. Trigger scan: - Row 3: P4=1 -> FIRES -> mark P3, P5 TRIGGER-INVALID - Row 7: P3=0 -> FIRES -> mark P5 (already), P6 TRIGGER-INVALID - All other rows: no match Trigger-invalid: P3, P5, P6 P1=VALID, P2=VALID, P3=INVALID, P4=VALID, P5=INVALID, P6=INVALID ** Output: 3 ** --- ** Example 4: ’729155’ ** P1=7, P2=2, P3=9, P4=1, P5=5, P6=5 Base: all valid. Trigger scan: - Row 3: P4=1 -> FIRES -> mark P3, P5 TRIGGER-INVALID - Row 9: P5=5 -> FIRES -> mark P2, P6 TRIGGER-INVALID - All other rows: no match Trigger-invalid: P2, P3, P5, P6 P1=VALID, P2=INVALID, P3=INVALID, P4=VALID, P5=INVALID, P6=INVALID ** Output: 2 ** --- ** Example 5: ’729489’ ** P1=7, P2=2, P3=9, P4=4, P5=8, P6=9 Base: all valid. Trigger scan: - Row 10: P5=8 -> FIRES -> mark P2, P6 TRIGGER-INVALID - Row 12: P6=9 -> FIRES -> mark P3, P5 TRIGGER-INVALID - All other rows: no match Trigger-invalid: P2, P3, P5, P6 35 P1=VALID, P2=INVALID, P3=INVALID, P4=VALID, P5=INVALID, P6=INVALID ** Output: 2 ** --- ** Example 6: ’129402’ ** P1=1, P2=2, P3=9, P4=4, P5=0, P6=2 Base: all valid. Trigger scan: - Row 2: P1=1 -> FIRES -> mark P3, P4 TRIGGER-INVALID - Row 11: P6=2 -> FIRES -> mark P3 (already), P5 TRIGGER-INVALID - All other rows: no match Trigger-invalid: P3, P4, P5 P1=VALID, P2=VALID, P3=INVALID, P4=INVALID, P5=INVALID, P6=VALID ** Output: 3 ** --- ** Example 7: ’789405’ ** P1=7, P2=8, P3=9, P4=4, P5=0, P6=5 Base: all valid. Trigger scan: - Row 6: P2=8 -> FIRES -> mark P3, P5 TRIGGER-INVALID - All other rows: no match Trigger-invalid: P3, P5 P1=VALID, P2=VALID, P3=INVALID, P4=VALID, P5=INVALID, P6=VALID ** Output: 4 ** --- ** Example 8: ’729705’ ** P1=7, P2=2, P3=9, P4=7, P5=0, P6=5 Base: all valid. Trigger scan: - Row 4: P4=7 -> FIRES -> mark P3, P6 TRIGGER-INVALID - All other rows: no match Trigger-invalid: P3, P6 P1=VALID, P2=VALID, P3=INVALID, P4=VALID, P5=VALID, P6=INVALID ** Output: 4 ** --- ** Example 9: ’723705’ ** P1=7, P2=2, P3=3, P4=7, P5=0, P6=5 Base: all valid. Trigger scan: - Row 4: P4=7 -> FIRES -> mark P3, P6 TRIGGER-INVALID - Row 8: P3=3 -> FIRES -> mark P5, P6 (already) TRIGGER-INVALID - All other rows: no match Trigger-invalid: P3, P5, P6 P1=VALID, P2=VALID, P3=INVALID, P4=VALID, P5=INVALID, P6=INVALID ** Output: 3 ** --- ** Example 10: ’769405’ ** P1=7, P2=6, P3=9, P4=4, P5=0, P6=5 Base: all valid. Trigger scan: - Row 5: P2=6 -> FIRES -> mark P3, P5 TRIGGER-INVALID 36 - All other rows: no match Trigger-invalid: P3, P5 P1=VALID, P2=VALID, P3=INVALID, P4=VALID, P5=INVALID, P6=VALID ** Output: 4 ** goemotions: conservative labeling and ordered rules. goemotions base program (instruction) Identify every emotion expressed in the text. A comment may express more than one emotion. Return an array drawn from the declared 28-label vocabulary. Include every category that clearly applies; use ["neutral"] only when no other emotion applies. The seed’s instruction to include every plausible emotion caused systematic over-labeling. At iter- ation 7, the curator queried the seed’s full failure set, inspected representative rows, and then used question memory to contextualize partial scores. For “Happy Cake day!” the gold label was joy but the parent received 0.5; for “This is my favorite wild memory” it received 0.4. These scores exposed a common mechanism: the parent was returning the correct emotion together with one or more unnecessary labels. The curator prescribed a conservative single-label default. The mutator’s first draft passed eight of ten selected rows; after adding rules for sarcastic positive language and social check-ins, the same live verifier reported ten of ten. The committed child increased the search objective from 0.0022 to 0.1741 in one iteration. The final champion generalizes this idea into a strict-priority rule system: goemotions champion (abridged) Classify emotions using the GoEmotions annotation scheme. ALLOWED LABELS --- use only the 28 labels in the declared vocabulary. Apply the following rules in strict priority order. Record which rule matched, then output only that rule’s prescribed emotion list. RULE A --- GRATITUDE EXPRESSION: If the text starts or ends with "thanks" or "thank you", output ["gratitude"] only. RULE B --- EXPLICIT FIRST-PERSON EMOTION: Check explicit worry, fear, love, sadness, joy, curiosity, pride, and relief patterns in order; stop at the first match. RULE C --- INTERPERSONAL STATE QUESTION: For a direct question about another person’s wellbeing, output ["caring", "curiosity"]. ... DEFAULT: Return the single most prominent emotion unless two emotions are independently and explicitly expressed. The champion was discovered at iteration 47 and obtains the held-out score 0.444 reported in Ta- ble 2. Although the artifact is prompt-heavy, its structure is algorithmic: ordered guards, explicit exceptions, a closed vocabulary, and a forced structured output replace the seed’s unconstrained semantic judgment. finqa and convfinqa: executable arithmetic contracts. 37 finqa/convfinqa base programs (instructions) FINQA: Answer a numerical question over a financial report (text + table). Work out the answer and return the final numeric value (or yes/no). Use no units, thousands separators, or percent sign. CONVFINQA: Answer the current turn of a multi-turn numerical conversation over a financial report. Use dialogue history to resolve references such as "the difference" or "this". Return the final plain numeric value (or yes/no), with no units, commas, or percent sign. The financial-QA seeds merely ask the model to work out a numeric answer. Their champions instead define an operational contract for reference resolution, formula selection, sign direction, units, and answer serialization. The finqa program distinguishes percent change, signed absolute change, relative increase, same-period differences, ratios, averages, CAGR, and yes/no questions. The conversational champion adds rules for resolving “that,” “the difference,” and “the last year,” and for reusing prior answers without silently substituting a nearby year. finqa champion (abridged) Work through each question step by step: 1. Identify the relevant numbers from the text or table. 2. Match the question’s exact wording to the correct formula. 3. Compute carefully and show intermediate steps. FORMULA SELECTION: "percent change" -> (new - old) / old; preserve the sign "change in X" -> new - old in the table’s stated unit "increase/decrease" -> (new - old) / old as a signed ratio "how much bigger" -> X - Y; do not divide "average" -> sum / count "ratio of X to Y" -> X / Y "CAGR" -> (end / start)ˆ(1 / years) - 1, then x100 Return plain values only: no currency signs, commas, units, or percent signs. Do not convert away from the table’s stated unit. Sign matters. convfinqa champion (abridged) STEP 1 --- RESOLVE REFERENCES: Resolve pronouns and shorthand from the dialogue history. If the current question requests a value already stated for the same quantity and year, reuse that exact value. STEP 2 --- IDENTIFY OPERATION AND SIGNS: "decline from A to B" -> A - B "change from A to B" -> B - A "combined total" -> sum the referenced values "as a percentage of" -> component / whole STEP 3 --- EXTRACT VALUES: List every required value and its source. Match the requested year exactly. STEP 4 --- COMPUTE: Show the arithmetic explicitly, then emit a plain numeric answer 38 with no unit, comma, or percent sign. These contracts address errors that are difficult to repair with isolated answer demonstrations: the same numbers can require a ratio, signed difference, or magnitude depending on a short phrase. The resulting held-out scores are 0.813 for finqa and 0.953 for convfinqa. multilexnorm: conservative token transformation. multilexnorm base program (instruction) Normalize noisy social-media text. Rewrite each non-standard token to its canonical form (e.g., u->you, tmrw->tomorrow, gonna->going to, dont->don’t). Leave standard words, @mentions, #hashtags, and URLs unchanged. Return the full normalized text. Exact-match normalization is unusually sensitive to helpful but unauthorized rewrites. The evolved champion therefore begins with preservation rules, then applies a bounded lexicon of high- confidence transformations while preserving capitalization, spacing, punctuation, mentions, hash- tags, URLs, proper nouns, and uncertain or non-English tokens. multilexnorm champion (abridged) Rewrite each non-standard English token to its canonical form, leaving everything else exactly as written. ALWAYS LEAVE UNCHANGED: - @mentions, #hashtags, URLs, proper nouns, and non-English words - already-standard informal forms such as wanna, kinda, lol, and btw - uncertain tokens; when in doubt, leave the token unchanged - emotional sound effects whose repeated letters encode emphasis CAPITALIZATION: Preserve the exact capitalization pattern of the original token. SPACING AND PUNCTUATION: Preserve all whitespace and punctuation spacing exactly. HIGH-CONFIDENCE TRANSFORMS: dont -> don’t; u -> you; ur -> your; pls/plz -> please; tryn/tryna -> trying to; thx/thnx -> thanks; nvm -> never mind. The final program also records exceptions for contractions, elongated forms, and truncated -ing words. This “transform only when confident” policy is the opposite of the seed’s broad invitation to normalize and achieves 0.587 on the held-out set, a 7.7 p gain over GEPA. lex glue: hierarchy of legal exceptions. lexglue base program (instruction) Classify a US Supreme Court opinion by its primary SCDB issue area. Choose exactly one of: Criminal Procedure, Civil Rights, First Amendment, Due Process, Privacy, Attorneys, Unions, Economic Activity, Judicial Power, Federalism, Interstate Relations, Federal Taxation, Miscellaneous. Use the label glossary: rights of the accused -> Criminal Procedure; discrimination and voting -> Civil Rights; speech, press, and religion -> First Amendment; abortion and personal autonomy -> Privacy; fees and professional discipline -> Attorneys; labor relations -> Unions; antitrust, contracts, torts, and business regulation -> Economic Activity; jurisdiction, standing, mootness, and remedies -> 39 Judicial Power. Choose the issue area that dominates the holding. The legal classifier evolves from a short glossary into a hierarchy of issue-area distinctions. Its champion begins with a priority scan for attorney-fee holdings, then separates the constitutional substance of a case from its procedural posture. For example, it distinguishes a speaker denied access to a government forum (First Amendment) from a captive listener resisting unwanted content (Privacy); federal-court jurisdiction over an agency order (Judicial Power) from the substantive scope of an agency’s economic authority (Economic Activity); and federal labor preemption (Federalism) from the underlying labor right (Unions). The complete prompt is long, but these repeated “default–exception–counterexample” clauses are precisely the structure that the archive accumulated from observed confusions. The champion scores 0.737 on held-out opinions. lexglue champion (abridged) Choose exactly one SCDB issue area from the declared 13-label vocabulary. FIRST --- ATTORNEY FEES PRIORITY SCAN: If the primary holding concerns attorney-fee entitlement, computation, recovery, or denial, classify as Attorneys immediately; the underlying substantive dispute does not change the classification. SPEAKER / LISTENER DISTINCTION: A speaker denied access to a government forum is First Amendment. A captive listener resisting compelled content is Privacy. AGENCY CASES: Whether a federal court may review an agency order is Judicial Power. The substantive scope of the agency’s regulatory authority is Economic Activity (or Unions for a substantive labor right). LABOR PREEMPTION: Federal labor law displacing state or local regulation is Federalism; the underlying organizing or bargaining right is Unions. Together, these artifacts show the same pattern at different scales: evolution replaces broad task descriptions with explicit precedence, exception, and serialization rules. HDATA ENGINEERING H.1DETAILED METHODOLOGY Data engineering optimization applies Algorithm 1 to data-eng-bench. We define 50 dev tasks and 53 held-out test tasks, randomly sampled from the full set of 103 tasks. Knowledge graphs. A primary output of an evolved pipeline is a knowledge graph (KG), pro- duced before the downstream agent attempts solving a task. The KG is a context-grounded mark- down briefing, KG.md, limited to 300 lines. Evolved KGs describe schemas, tables, columns, dbt models, outputs, and metrics. They also record joins, cardinalities, source()/ref() lineage, model dependencies, derivations, model grain, business rules, boundary behavior, null handling, fanout risks, validation checks, and task-specific pitfalls. The KG is not an RDF store and contains no solution SQL. The authoring pipeline verifies physi- cal table locations, column types, categorical values, null encodings, and row counts using SHOW, DESCRIBE, INFORMATION SCHEMA, and sqlexecute. This avoids relying on a potentially stale sources.yml file. 40 The evolved program. The evolved artifact is the Python dictionaryPIPELINESPEC in initial program.py. It jointly controls the KG authoring protocols, agent orchestration, and adherence test authoring protocol. Executing this specification produces the task-specific KG.md and test kg.py supplied to the downstream coding agent. The mutator can improve this pipeline through the following three components. 1. KG authoring protocols. The skills catalog describes reusable procedures for constructing different parts of the KG, including its usage guide, metrics and busi- ness rules, and warehouse and model structure.Catalog entries contain metadata rather than complete skill bodies. During execution, assigned agents expand them into runtime skills/<name>.md. These skills guide KG production and may be shared among agents in the authoring pipeline. 2. Agent orchestration. The agents[] list defines an ordered sequence of agent sessions that produce the KG. Each session has an editable prompt and a set of assigned skills. The mutator may rewrite prompts, add or remove agents, split or merge roles, reorder the pipeline, and redistribute skills. The agents share a sanitized workspace. This permits either a single agent or a pipeline with separate exploration, verification, and synthesis roles. 3. Adherence test protocol. A second form of context for the solver agent is executable tests derived from the KG. It is a protocol used during compilation rather than a section of KG.md. A fixed test authoring model consumes the KG together with this evolved protocol and produces test kg.py. Evolution can therefore improve both the declarative guid- ance given to the solver and the way that guidance is translated into executable feedback. Inference and evaluation. At inference time, the best evolved program is fixed and its executor materializes the evolved protocols and agent configuration. All evaluated agent roles use the same out-of-the-box claude-sonnet-5 backbone and harness. These roles include the ordered agent sessions that construct the KG and the downstream agent that consumes it before executing a task. Only KG.md is registered as a skill for the Harbor agent. The internal runtime skills/ * .md files are not forwarded. The sole model exception is the fixed adherence test author, which uses claude-opus-4-8 with temperature 0.2 and a completion limit of 50,000 tokens. It translates KG.md and the evolved adherence test authoring protocol into testkg.py. We hard pin this model because claude-sonnet-5 frequently produced truncated or syntactically invalid long test suites, partic- ularly unterminated strings. The generated suite is supplied to the Harbor agent as self-correction feedback. It is not the hidden grader, which remains fixed throughout evaluation. We note that adherence tests were generated for these evaluation runs but were not used for all runs. We therefore treat adherence testing as optional additional context rather than a uniform component of the reported results. Search methods. The reported CoCoEvolve result uses the best program available after 200 iter- ations. We configure one island, an active population of 15 candidates, an archive capacity of 8, 8 parallel evaluations, and the P0/P1/P2 evolution graph over the dev pool. We compare against faith- ful ports of GEPA, Meta-Harness, and AdaEvolve using the same claude-sonnet-5 backbone, downstream agent, equivalent iteration budget, and seed program PIPELINESPEC. H.2EXAMPLES OF BEST PROGRAMS H.2.1EVOLVED PROGRAMS The evolved specification p ∗ retains the initial three seed skills and authors an additional four. Two are pre-exploration environment checks: kg-environment-preflight confirms a usable, au- thorized warehouse, and kg-dbt-profiles-default-path pins the ̃/.dbt/ default so fixes are persisted rather than session-local. The other two are a column-verification protocol requir- ing DESCRIBE evidence for every named column, and a worked-example protocol for weighted or ratio-based metrics. The kg-adherence-test-author description also tightens, requir- ing each stated business rule to be checked by live-SQL recomputation against source and banning 41 negative-only assertions. In this example, the initial architect agent is the sole agent; no other agents were spawned. Initial pipeline specificationp 0 _PIPELINE_SPEC = "skills": "kg-agent-usage-guide": "description": ( "Author the Agent Usage Guide: what the downstream solver must " "produce (object names, grain) and how to know it is done. " "Generic guidance only --- no task-specific table or column names." ), , "kg-metrics-and-business-rules": "description": ( "Author Metrics & Business Rules: record numeric thresholds, " "windows, and formulas from the instruction. Do not invent values." ), , "kg-structure": "description": ( "Author Structure: confirmed sources (live warehouse location), " "required models/outputs (name, grain, columns, dependencies), " "and joins (keys, cardinality). Verified warehouse facts only --- " "do not restate the instruction. Generic guidance only --- no " "task-specific table or column names." ), , # Compile-time only (platform emits test_kg.py after KG.md --- not a KG.md section). "kg-adherence-test-author": "description": ( "One precise [KG-ADHERENCE] test per testable KG fact; failure " "messages quote the KG. Ban generic smoke tests." ), , , # Seed topology: one architect (look + write in the same session so # query results stay in context). Split only if explore_coverage shows # the look was skipped and a handoff would help. "agents": [ "name": "architect", "prompt": ( "You architect the Knowledge Graph for task task_name (database " "db_name). Budget: timeout_sec:.0fs. " "The instruction is a brief, not the warehouse. Do not paraphrase " 42 "it into KG.md --- that briefing is useless to the solver. Looking " "is not optional. Use ‘sql_execute‘ in THIS session before you " "write; you are not done until every table you will name has been " "DESCRIBE (or SHOW COLUMNS). Do not stop at SHOW DATABASES. " "Confirm each source’s actual database.schema.table via SHOW / " "INFORMATION_SCHEMA. Do not stop because the instruction looks " "complete. For categorical / boolean / dirty-typed columns that " "feed a filter, join, or metric, GROUP BY / COUNT stored tokens, " "null encoding, and row counts. Name only what those queries " "confirmed. Verified " "facts only; no solution SQL. Then write ‘KG.md‘ (<=300 lines, " "first line ‘# Knowledge Graph: task_name‘). " "Structure KG.md by following EVERY skill listed under \"Skills to " "author\" below. Author each as ‘runtime_skills/<name >.md‘ before " "you finish." ), "skills": [ "kg-agent-usage-guide", "kg-metrics-and-business-rules", "kg-structure", ], , ], Best evolved pipeline specificationp ∗ _PIPELINE_SPEC = "skills": "kg-agent-usage-guide": "description": ( "Author the Agent Usage Guide: what the downstream solver must " "produce (object names, grain) and how to know it is done. " "Generic guidance only | no task-specific table or column names." ), , "kg-metrics-and-business-rules": "description": ( "Author Metrics & Business Rules: record numeric thresholds, " "windows, and formulas from the instruction. Do not invent values." ), , "kg-structure": "description": ( 43 "Author Structure: confirmed sources (live warehouse location), " "required models/outputs (name, grain, columns, dependencies), " "and joins (keys, cardinality). Verified warehouse facts only | " "do not restate the instruction. Generic guidance only | no " "task-specific table or column names." ), , # Compile-time only (platform emits test_kg.py after KG.md | not a KG.md section). "kg-adherence-test-author": "description": ( "One precise [KG-ADHERENCE] test per testable KG fact; failure " "messages quote the KG. Ban generic smoke tests. For any stated " "filter/dedup/exclusion business rule | especially a KG value " "explicitly labeled evidence rather than a prescribed target | " "the assertion must be a live SQL comparison: recompute the " "correct aggregate from source inside the test by embedding the " "exact stated predicate/dedup key, then assert the model output " "matches that live-recomputed value within a tight numeric " "tolerance. Do not assert only ’output != one known- wrong value’ " "for a business rule | a negative-only check can pass while the " "rule is still violated a different way; it must be paired with " "a positive, rule-derived comparison." ), , "kg-column-verification-protocol": "description": ( "Before naming any specific column in Structure or Metrics & " "Business Rules, that column must have an associated " "DESCRIBE/SHOW COLUMNS (or information_schema.columns) result " "from THIS session. If a column was not schema-probed, omit it " "or mark it explicitly unverified rather than stating it as " "fact. Generic guidance only | no task-specific table or " "column names." ), , "kg-environment-preflight": "description": ( "Before other exploration, verify the actual database session " "context the downstream solver will inherit: confirm which " 44 "warehouse is active/authorized for the working role ( a session " "may start with no active warehouse, or a default warehouse name " "that does not exist or is not authorized). Record the " "confirmed-working warehouse and the exact persistent fix (e.g. " "profile/project-level config or an explicit USE WAREHOUSE in the " "model-build path) as an Agent Usage Guide fact | not a " "session-local env-file edit a later step could silently revert. " "Generic guidance only; no task-specific table or column names, " "and no assumption about which warehouse name is broken vs " "working in a given account." ), , "kg-dbt-profiles-default-path": "description": ( "Author a profiles-directory fact for the Agent Usage Guide when " "the task requires creating a new dbt project (rather than " "editing a pre-existing one): dbt’s CLI defaults to " "‘ ̃/.dbt/profiles.yml‘ whenever ‘--profiles-dir‘/" "‘DBT_PROFILES_DIR‘ is not explicitly set on a given invocation. " "If the solver only writes ‘profiles.yml‘ to a custom path and " "relies on exporting ‘DBT_PROFILES_DIR‘ for its own manual ‘dbt " "run‘ calls, any OTHER process that later calls ‘dbt run‘/‘dbt " "build‘ in a fresh shell without that env var will fail at the " "default location instead. State explicitly: also place (or " "symlink) a copy of ‘profiles.yml‘ at the OS-default dbt " "profiles directory, not only the custom one. Verify by checking " "whether ‘ ̃/.dbt/‘ already exists and what it contains before " "assuming a custom path is sufficient. Generic dbt-CLI behavior " "| no task-specific table or column names." ), , "kg-composite-formula-worked-example": "description": ( "For any metric in Metrics & Business Rules that involves a " "weighted composite score, a percentile/rank direction " "(ascending vs. inverted), or a ratio-to-median/cap- and-invert " "normalization, do not describe the formula in prose alone. " 45 "Pick 1-2 real rows from the live warehouse, compute every " "intermediate factor by hand following the exact stated formula " "(including cap/invert/boundary behavior and NULL- fallback " "substitution), and record the final numeric result as a worked " "example in the KG. If the instruction’s wording is genuinely " "ambiguous even after this exercise, state the ambiguity " "explicitly as an open question rather than silently picking one " "interpretation and presenting it as fact. Never invent " "thresholds not present in the instruction. Generic guidance " "only | no task-specific table/column names or copied assertion " "values." ), , , # Seed topology: one architect (look + write in the same session so # query results stay in context). Split only if explore_coverage shows # the look was skipped and a handoff would help. "agents": [ "name": "architect", "prompt": ( "You architect the Knowledge Graph for task task_name (database " "db_name). Budget: timeout_sec:.0fs. " "The instruction is a brief, not the warehouse. Do not paraphrase " "it into KG.md | that briefing is useless to the solver. Looking " "is not optional. Use ‘sql_execute‘ in THIS session before you " "write; you are not done until every table you will name has been " "DESCRIBE (or SHOW COLUMNS). Do not stop at SHOW DATABASES. " "Confirm each source’s actual database.schema.table via SHOW / " "INFORMATION_SCHEMA. Do not stop because the instruction looks " "complete. For categorical / boolean / dirty-typed columns that " "feed a filter, join, or metric, GROUP BY / COUNT stored tokens, " "null encoding, and row counts. Name only what those queries " "confirmed. Verified " "facts only; no solution SQL. Then write ‘KG.md‘ (≤300 lines, " "first line ‘# Knowledge Graph: task_name‘). " "Structure KG.md by following EVERY skill listed under \"Skills to " 46 "author\" below. Author each as ‘runtime_skills/<name >.md‘ before " "you finish. " "Before any schema exploration, run the environment- preflight " "check: confirm the session actually has a usable, authorized " "warehouse (do not assume the default env/session warehouse " "works | it may not exist or may be unauthorized). Also confirm " "the dbt profiles-directory resolution fact from " "kg-dbt-profiles-default-path when the task builds a new dbt " "project. Fold both the confirmed-working warehouse/ persistent " "fix and the profiles-directory fact into the Agent Usage Guide " "section of KG.md; do not add new top-level sections for them " "and do not exceed the 300-line budget. " "For any weighted composite score or percentile/ratio- based " "metric in Metrics & Business Rules, follow " "kg-composite-formula-worked-example: compute at least one " "worked numeric example from live rows before writing the " "formula down, rather than describing it in prose alone." ), "skills": [ "kg-agent-usage-guide", "kg-metrics-and-business-rules", "kg-structure", "kg-environment-preflight", "kg-column-verification-protocol", "kg-dbt-profiles-default-path", "kg-composite-formula-worked-example", ], , ], H.2.2FINAL PIPELINE SPECIFICATION ARTIFACTS Below is KG.md for task dbt-fix-marketing-attribution, produced by p ∗ . It records verified tables, columns, data values, and environment settings for the downstream solver agent. It also flags differences between the task instruction and the live warehouse instead of making unsup- ported assumptions. Final KG.md for dbt-fix-marketing-attribution produced by pipeline specificationp ∗ # Knowledge Graph: dbt-fix-marketing-attribution ## Structure ** Confirmed sources ** (database ‘DBT_BENCH_RETAIL‘, schema ‘MAIN‘; all confirmed via ‘SHOW TABLES‘/‘SHOW VIEWS‘/‘INFORMATION_SCHEMA. TABLES‘ + ‘DESCRIBE VIEW‘): 47 - ‘DBT_BENCH_RETAIL.MAIN.STG_GA__SESSIONS‘ (VIEW, 5053 rows, 5053 distinct ‘SESSION_ID‘, 0 NULL ‘SESSION_ID‘/‘SESSION_START‘) Columns: ‘SESSION_ID‘, ‘VISITOR_ID‘, ‘CUSTOMER_ID‘, ‘CHANNEL_ID‘, ‘ SESSION_START‘ (VARCHAR | timestamp-like string, NOT native date/ timestamp), ‘SESSION_END‘ (VARCHAR), ‘DURATION_SECONDS‘ (NUMBER), ‘PAGE_VIEWS‘ (NUMBER), ‘LANDING_PAGE‘, ‘EXIT_PAGE‘, ‘REFERRER‘, ‘ UTM_SOURCE‘, ‘UTM_MEDIUM‘, ‘UTM_CAMPAIGN‘, ‘DEVICE_TYPE‘, ‘BROWSER ‘, ‘OS‘, ‘IP_ADDRESS‘, ‘COUNTRY‘, ‘IS_CONVERTED‘ (VARCHAR), ‘ ORDER_ID‘ (VARCHAR, always NULL in this env | 0/5053 populated), ‘ CREATED_AT‘, plus metadata cols ‘_LOADED_AT‘, ‘_SOURCE_SYSTEM‘, ‘ _BATCH_ID‘, ‘_ROW_NUMBER‘, ‘_ROW_HASH‘. - ‘DBT_BENCH_RETAIL.MAIN.STG_GA__EVENTS‘ (VIEW, 15047 rows) Columns: ‘EVENT_ID‘, ‘SESSION_ID‘ (1131 rows NULL), ‘EVENT_TYPE‘, ‘ EVENT_NAME‘, ‘EVENT_TIMESTAMP‘ (VARCHAR; 0 literal NULLs but 3801/15047 rows fail ‘TRY_CAST(... AS TIMESTAMP)‘ | dirty/ malformed timestamp strings), ‘PAGE_URL‘, ‘ELEMENT_ID‘, ‘ ELEMENT_CLASS‘, ‘PRODUCT_ID‘, ‘EVENT_VALUE‘ (VARCHAR storing numeric-looking decimal strings, e.g. ‘86.52592478731349‘), ‘ EVENT_DATA‘, ‘CREATED_AT‘, plus same metadata cols as sessions. - ‘DBT_BENCH_RETAIL.MAIN.INT_SESSIONS_EVENTS_JOINED‘ (VIEW, LEFT JOIN of the two above; 15775 rows vs 5053 distinct ‘SESSION_ID‘ → confirmed fanout ̃3.1x rows per session) Columns (15 total, confirmed via ‘DESCRIBE VIEW‘): ‘SESSION_ID‘, ‘ VISITOR_ID‘, ‘CUSTOMER_ID‘, ‘CHANNEL_ID‘, ‘SESSION_START‘, ‘ SESSION_END‘, ‘DURATION_SECONDS‘, ‘PAGE_VIEWS‘, ‘T2_EVENT_ID‘, ‘ T2_EVENT_TYPE‘, ‘T2_EVENT_NAME‘, ‘T2_EVENT_TIMESTAMP‘, ‘ T2_PAGE_URL‘, ‘T2_ELEMENT_ID‘, ‘T2_ELEMENT_CLASS‘. ** ‘T2_EVENT_VALUE‘ DOES NOT EXIST in this view in this environment ** | confirmed absent from the 15-column ‘DESCRIBE VIEW‘ output. This matches the instruction’s warned "may omit ‘t2_event_value‘ entirely" branch, and in this live warehouse that branch is the actual case: revenue cannot be sourced from ‘ int_sessions_events_joined‘ here. Rejected/decoy candidates seen in schema search (do not use | wrong grain or duplicative of the above): ‘STG_RAW_GA__SESSIONS * ‘, ‘ STG_SESSIONS * ‘, ‘RAW_GA.SESSIONS * ‘, ‘STG_RAW_GA__EVENTS * ‘, ‘ FCT_MULTI_TOUCH_ATTRIBUTION‘, ‘FCT_RECONSTRUCTED_SESSIONS‘, ‘ RPT_SESSION_HOURLY_DISTRIBUTION‘ | these are separate staging- history/rebuild variants and pre-existing marts unrelated to the required ‘int_sessions_events_joined‘ / ‘stg_ga__sessions‘ / ‘ stg_ga__events‘ lineage named by the instruction. ** Required output ** (per instruction, not re-verified in warehouse since it does not yet exist): dbt model ‘marts/marketing/ rpt_attribution_fixed.sql‘, materializing to schema ‘main‘ ( Snowflake backend) or ‘analytics‘ (DuckDB backend). Grain: one row per (‘attribution_date‘, ‘channel‘, ‘medium‘, ‘campaign‘). Depends on ‘stg_ga__sessions‘, ‘stg_ga__events‘, ‘ int_sessions_events_joined‘ (all confirmed above). ** Joins: ** - ‘stg_ga__sessions.SESSION_ID‘ = ‘int_sessions_events_joined. SESSION_ID‘ | this is the join already baked into the ‘ int_sessions_events_joined‘ view (LEFT JOIN sessions→events); cardinality 1:many, fanout confirmed 5053 sessions → 15775 joined rows. - ‘stg_ga__events.SESSION_ID‘ = ‘stg_ga__sessions.SESSION_ID‘ | 1131/15047 event rows have NULL ‘SESSION_ID‘ (orphan events, cannot join). 48 ## Metrics & Business Rules See ‘runtime_skills/kg-metrics-and-business-rules.md‘ for the generic capture protocol. Task-specific verified facts: ** Windows: ** 90-day trailing window anchored to ‘MAX(CAST( session_start AS DATE))‘ computed from ‘stg_ga__sessions‘ itself ( per instruction) | NOT anchored to current wall-clock date. Confirmed ‘SESSION_START‘ values in ‘stg_ga__sessions‘ range from ‘1765-02-04‘ to ‘2026-01-03‘; 46/5053 rows are pre-2020 outlier/ garbage dates, 5007/5053 fall in a normal recent range. The MAX used for the 90-day anchor will be pulled from this same distribution (max observed = ‘2026-01-03 19:00:08‘). ** Formula: ** ‘conversion_rate = conversions / sessions‘, divide-by- zero must be handled (per instruction, ‘NULLIF‘-style guard), rounded to 4 decimals. ** Default/mapping rules (verified against actual NULL counts): ** - ‘UTM_SOURCE‘ NULL → ‘’direct’‘: 387/5053 sessions have NULL ‘ UTM_SOURCE‘. - ‘UTM_MEDIUM‘ NULL → ‘’none’‘: 402/5053 sessions have NULL ‘ UTM_MEDIUM‘. - ‘UTM_CAMPAIGN‘ NULL → ‘’none’‘: 3663/5053 sessions have NULL ‘ UTM_CAMPAIGN‘ (majority of sessions are uncampaigned). ** Fallback/precedence rule (verified, not assumed): ** Instruction says prefer ‘t2_event_value‘ from ‘int_sessions_events_joined‘, else use ‘event_value‘ from ‘stg_ga__events‘. In this environment, ‘ t2_event_value‘ is confirmed absent (see Structure section) | the fallback branch to ‘stg_ga__events.EVENT_VALUE‘ is the one that applies here. ** Dedup/grain rule: ** Instruction requires summing revenue per session first, then aggregating to the final (date, channel, medium, campaign) grain, to avoid inflation from the confirmed 1: ̃3.1 session→event-row fanout in ‘int_sessions_events_joined‘. ** Verified discrepancies (facts only, not resolved here): ** - Instruction’s conversion rule states ‘is_converted = 1‘. Actual stored tokens in ‘STG_GA__SESSIONS.IS_CONVERTED‘ (VARCHAR column, confirmed via ‘GROUP BY‘): ‘’0’‘=1532, ‘’1’‘=47, ‘’N’‘=997, ‘’Y ’‘=35, ‘’false’‘=2364, ‘’true’‘=78 (total 5053). Truthy-token candidates (‘’1’‘,‘’Y’‘,‘’true’‘) sum to 160 sessions ( ̃3.2%); falsy-token candidates (‘’0’‘,‘’N’‘,‘’false’‘) sum to 4893. A literal ‘= 1‘ comparison against this VARCHAR column will not match the ‘’Y’‘/‘’true’‘ rows. - Instruction’s revenue/conversion-event rule states ‘event_name = ’ purchase’‘ (and ‘t2_event_name = ’purchase’‘). A case-insensitive scan (‘ILIKE ’%purchas%’‘) across all 15047 rows of ‘ STG_GA__EVENTS.EVENT_NAME‘ returned ** 0 matching rows ** . Distinct ‘EVENT_NAME‘ values actually present (51 distinct, 1216 rows literal NULL): ‘’View Product’‘(1420), ‘’Search’‘(1418), ‘’Filter ’‘(1410), ‘’Video Play’‘(1406), ‘’Click’‘(1391), ‘’Form Submit ’‘(1384), ‘’Add To Cart’‘(1357), ‘’Sort’‘(1344), ‘’Scroll’‘(1333), ‘’Remove From Cart’‘(1321), plus ̃40 additional low-frequency dirty tokens of the form ‘’event_N.0’‘ (each count 12, ̃46 rows total). The same absence of ‘’purchase’‘ holds in ‘ INT_SESSIONS_EVENTS_JOINED.T2_EVENT_NAME‘ (11 distinct groups observed, same 10 named categories plus NULL, no ‘’purchase’‘). - ‘STG_GA__EVENTS.EVENT_VALUE‘ is stored as VARCHAR containing decimal numeric strings (e.g. ‘’86.52592478731349’‘) | requires explicit cast to a numeric type before summing. 49 - ‘SESSION_START‘, ‘SESSION_END‘, ‘EVENT_TIMESTAMP‘ are all stored as VARCHAR, not native DATE/TIMESTAMP, in this environment | confirmed via ‘DESCRIBE VIEW‘. ‘TRY_CAST(SESSION_START AS TIMESTAMP)‘ succeeded on sampled rows; ‘TRY_CAST(EVENT_TIMESTAMP AS TIMESTAMP)‘ failed on 3801/15047 rows (dirty timestamp strings in the events table specifically, not the sessions table). - ‘STG_GA__SESSIONS.CUSTOMER_ID‘ is NULL on 3632/5053 rows (guest/ anonymous sessions) and ‘ORDER_ID‘ is NULL on all 5053 rows in this environment. ## Environment & Execution Notes ** Fresh-session, zero-override check: ** ‘SELECT CURRENT_WAREHOUSE(), CURRENT_ROLE(), CURRENT_DATABASE(), CURRENT_SCHEMA(), CURRENT_USER (), CURRENT_ACCOUNT()‘ run in a brand-new ‘sql_execute‘ session with no prior ‘USE ...‘ statements issued in that session and no override of the architect’s or any prior agent’s context → literal result: ‘WH=KG_AUTHOR_WH, ROLE=PUBLIC, DB=DBT_BENCH_RETAIL , SCHEMA=MAIN, USER=DEX_EVAL_USER, ACCT=UOB68132‘ (query_id ‘01 c72a54-030b-8aae-0032-d0c3016224b6‘). The default connection resolved cleanly | no NULL/none, no failure to record or fix. ** Compute verification: ** ‘SELECT COUNT( * ) FROM DBT_BENCH_RETAIL.MAIN. STG_GA__SESSIONS‘ on that same resolved context → ‘5053‘ rows returned successfully (query_id ‘01c72a54-030b-8aae-0032- d0c3016224e6‘). This confirms ‘KG_AUTHOR_WH‘ is not just a name returned by ‘CURRENT_WAREHOUSE()‘ | it is actually authorized and usable by role ‘PUBLIC‘ for real compute in this environment. ** Schema alignment (checked, not assumed): ** The task instruction requires the Snowflake output model to land in schema ‘main‘ via a ‘generate_schema_name‘ macro override. The live session’s ‘ CURRENT_SCHEMA()‘ already resolves to ‘MAIN‘ with zero overrides. These agree in this environment | there is no live discrepancy between the instruction’s hardcoded schema literal and the session ’s default schema to flag here. ** Instruction to downstream solver: ** No environment fix was needed to produce this KG | the default warehouse/role/database/schema all resolved and worked with zero manual overrides. If, in your own session, you find you must override any of these (e.g. to get ‘dbt run‘ or your own self-tests to pass), leave that override in the ** persisted ** profile/environment used for final scoring (e.g. ‘ profiles.yml‘, environment variables, or whatever config file supplies these values at grading time) | do NOT revert it back afterward under the belief that reverting is "cleanup." The independent grading connection opens its own fresh session and inherits only the persisted defaults, not anything you fixed transiently in your live session. Immediately before finishing, verify with a second, genuinely fresh connection/session (not the one you just patched) that the build still succeeds under the exact environment state that will be graded. Do not blanket- override the schema/database literal in ‘profiles.yml‘/macros based solely on this session’s env var reading | the alignment check above already confirms they agree in this environment, so no override is warranted here without new evidence. ## Known Ambiguities & Risk Flags - ** ‘is_converted‘ truthy definition ** (real COUNT queries against ‘ DBT_BENCH_RETAIL.MAIN.STG_GA__SESSIONS‘, 5053 total rows): Instruction states a session is converted if ‘is_converted = 1‘. Interpretation A | literal ‘IS_CONVERTED = ’1’‘ → ** 47 ** sessions 50 . Interpretation B | truthy-token set ‘IS_CONVERTED IN (’1’,’Y’,’ true’)‘ → ** 160 ** sessions (‘’1’‘=47, ‘’Y’‘=35, ‘’true’‘=78). Complementary falsy-token set ‘IS_CONVERTED IN (’0’,’N’,’false’)‘ → ** 4893 ** sessions (‘’0’‘=1532, ‘’N’‘=997, ‘’false’‘=2364). Disagreement: 47 vs 160 | a ̃3.4x difference in how many sessions count as converted. Affects: ‘conversions‘ numerator, ‘ conversion_rate‘, and which sessions’ revenue is eligible to be summed. Not resolved here | solver must choose an interpretation and be prepared to justify it; both counts are real and verified, not modeled. - ** ‘event_name = ’purchase’‘ literal match ** (real COUNT query against ‘DBT_BENCH_RETAIL.MAIN.STG_GA__EVENTS‘, 15047 total rows, 1216 NULL ‘EVENT_NAME‘): Instruction states conversion/revenue events are identified by ‘event_name = ’purchase’‘ (or ‘ t2_event_name = ’purchase’‘ in the joined view). Literal match ‘ EVENT_NAME = ’purchase’‘ → ** 0 ** rows. Case-insensitive fuzzy match ‘EVENT_NAME ILIKE ’%purchas%’‘ → ** 0 ** rows. Distinct-value scan confirms 51 distinct non-NULL ‘EVENT_NAME‘ values actually present, none resembling purchase (top values: ‘’View Product ’‘=1420, ‘’Search’‘=1418, ‘’Filter’‘=1410, ‘’Video Play’‘=1406, ‘’ Click’‘=1391, ‘’Form Submit’‘=1384, ‘’Add To Cart’‘=1357, ‘’Sort ’‘=1344, ‘’Scroll’‘=1333, ‘’Remove From Cart’‘=1321, plus ̃40 low- frequency dirty ‘’event_N.0’‘ tokens). Risk: under the literal instruction rule, ‘attributed_revenue‘ will be ** 0 for every row ** of the final output, because the named conversion-event literal does not exist anywhere in this environment’s data under any casing tried. This is a structural risk, not a two-way count disagreement | flagging it rather than substituting a different event-name literal on the solver’s behalf. Runtime skill files authored per assignment (full protocols, generic/ reusable, no task-specific names inside them): - ‘runtime_skills/kg-agent-usage-guide.md‘ - ‘runtime_skills/kg-metrics-and-business-rules.md‘ - ‘runtime_skills/kg-structure.md‘ - ‘runtime_skills/kg-ambiguity-guardrails.md‘ - ‘runtime_skills/kg-environment-handoff.md‘ The suite below turns KG statements into executable checks. Some tests verify recorded facts against the live data warehouse, while others recompute business rules from source data and compare them with the solver’s output. Their failure messages point the solver to the relevant KG statement, helping it locate and fix mismatches. Adherence test suite testkg.py authored by claude-opus-4-8 """Self-correction test suite for task: dbt-fix-marketing-attribution. [KG-ADHERENCE] tests encode specific facts stated in the Knowledge Graph (KG): required output model, schema, grain, columns, metric formulas, filter predicates, UTM default mappings, dedup-before-aggregate rules, and the confirmed numeric anchors (160 truthy is_converted sessions, 1216 NULL event_name rows, 5053 sessions). [UNIVERSAL] tests encode output-quality invariants the KG’s Output requirements imply (no NULL required columns, conversion_rate in [0,1], conversions <= sessions) and cross-source reconciliation against the confirmed sources. 51 These tests are derived EXCLUSIVELY from the KG | not from instruction .md, not from any solution SQL, and not from the hidden grader. Passing them is strong evidence the agent followed the KG. A failing test points to a specific KG sentence to re-read (quoted in each failure message). """ import os import sys _here = os.path.dirname(os.path.abspath(__file__)) for _p in [_here, os.path.normpath(os.path.join(_here, "..", ".."))]: if os.path.exists(os.path.join(_p, "test_backbone.py")): sys.path.insert(0, _p) break from test_backbone import execute_query, execute_scalar, make_conn_fixture, require # -- Task constants -------------------------------------------------- ---------- AGENT_PROJECT_DIR = "/app/dbt_project" # Output schema: main (Snowflake) or analytics (DuckDB). We resolve per db_type. SCHEMA_DUCKDB = "analytics" SCHEMA_SNOWFLAKE = "main" MODEL = "rpt_attribution_fixed" MODEL_FILE = f"AGENT_PROJECT_DIR/models/marts/marketing/ rpt_attribution_fixed.sql" # Confirmed sources (database DBT_BENCH_RETAIL, schema MAIN). SRC_SCHEMA = "main" STG_SESSIONS = "stg_ga__sessions" STG_EVENTS = "stg_ga__events" INT_JOINED = "int_sessions_events_joined" # Expected grain columns per KG ("one row per (attribution_date, channel, medium, campaign)"). GRAIN_COLS = ["attribution_date", "channel", "medium", "campaign"] conn = make_conn_fixture(check_paths=[MODEL_FILE]) def _out_schema(db_type): return SCHEMA_DUCKDB if db_type == "duckdb" else SCHEMA_SNOWFLAKE def _fqn(schema, table): return schema + "." + table def _columns(c, db_type, schema, table): if db_type == "duckdb": rows = execute_query( c, db_type, "SELECT name FROM pragma_table_info(’" + schema + "." + table + "’)", ) return set(str(r[0]).lower() for r in rows) 52 rows = execute_query( c, db_type, "SELECT column_name FROM information_schema.columns " "WHERE lower(table_schema) = lower(’" + schema + "’) " "AND lower(table_name) = lower(’" + table + "’)", ) return set(str(r[0]).lower() for r in rows) # -- Prerequisite ---------------------------------------------------- -------- def test_dbt_run_prerequisite(conn): c, db_type = conn schema = _out_schema(db_type) cnt = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(schema, MODEL), ) require( cnt is not None and cnt > 0, "[KG-ADHERENCE] Required output dbt model ’marts/marketing/" "rpt_attribution_fixed.sql’ must be built and populated | run dbt first.", ) # -- Source sanity anchors (confirmed in KG) ------------------------- -------- def test_source_sessions_row_count(conn): c, db_type = conn cnt = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS), ) require( cnt == 5053, "[UNIVERSAL] KG: ’STG_GA__SESSIONS (VIEW, 5053 rows)’.", ) def test_source_sessions_distinct_session_id(conn): c, db_type = conn cnt = execute_scalar( c, db_type, "SELECT COUNT(DISTINCT session_id) FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS), ) require( cnt == 5053, "[UNIVERSAL] KG: ’5053 distinct SESSION_ID, 0 NULL SESSION_ID ’.", ) def test_source_events_row_count(conn): c, db_type = conn cnt = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, STG_EVENTS), ) require( cnt == 15047, "[UNIVERSAL] KG: ’STG_GA__EVENTS (VIEW, 15047 rows)’.", 53 ) def test_source_int_joined_fanout(conn): c, db_type = conn total = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, INT_JOINED), ) distinct = execute_scalar( c, db_type, "SELECT COUNT(DISTINCT session_id) FROM " + _fqn(SRC_SCHEMA, INT_JOINED), ) require( total == 15775 and distinct == 5053, "[KG-ADHERENCE] KG: ’INT_SESSIONS_EVENTS_JOINED ... 15775 rows vs 5053 " "distinct SESSION_ID -> fanout ̃3.1x rows per session’. This fanout is " "why revenue must be summed per session BEFORE final aggregation.", ) def test_int_joined_lacks_t2_event_value(conn): c, db_type = conn cols = _columns(c, db_type, SRC_SCHEMA, INT_JOINED) require( "t2_event_value" not in cols, "[KG-ADHERENCE] KG: ’T2_EVENT_VALUE DOES NOT EXIST in this view in this " "environment’ | the fallback to stg_ga__events.EVENT_VALUE applies here.", ) # -- is_converted truthy anchor (160) -------------------------------- --------- def test_is_converted_truthy_anchor_160(conn): c, db_type = conn cnt = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS) + " WHERE is_converted IN (’1’,’Y’,’true’)", ) require( cnt == 160, "[KG-ADHERENCE] KG: truthy-token set IS_CONVERTED IN (’1’,’Y ’,’true’) " "-> 160 sessions (’1’=47,’Y’=35,’true’=78).", ) def test_is_converted_literal_one_anchor_47(conn): c, db_type = conn cnt = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS) + " WHERE is_converted = ’1’", ) require( cnt == 47, 54 "[KG-ADHERENCE] KG: literal IS_CONVERTED = ’1’ -> 47 sessions " "(Interpretation A of the is_converted ambiguity).", ) def test_is_converted_falsy_anchor_4893(conn): c, db_type = conn cnt = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS) + " WHERE is_converted IN (’0’,’N’,’false’)", ) require( cnt == 4893, "[KG-ADHERENCE] KG: falsy-token set IS_CONVERTED IN (’0’,’N’,’ false’) " "-> 4893 sessions.", ) # -- event_name = ’purchase’ structural risk ------------------------- --------- def test_event_name_null_anchor_1216(conn): c, db_type = conn cnt = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, STG_EVENTS) + " WHERE event_name IS NULL", ) require( cnt == 1216, "[KG-ADHERENCE] KG: ’STG_GA__EVENTS ... 1216 NULL EVENT_NAME ’.", ) def test_event_name_purchase_absent(conn): c, db_type = conn cnt = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, STG_EVENTS) + " WHERE event_name = ’purchase’", ) require( cnt == 0, "[KG-ADHERENCE] KG: literal EVENT_NAME = ’purchase’ -> 0 rows. " "Structural risk: attributed_revenue may be 0 for every output row.", ) # -- UTM default-mapping NULL anchors -------------------------------- --------- def test_utm_source_null_anchor_387(conn): c, db_type = conn cnt = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS) + " WHERE utm_source IS NULL", ) require( 55 cnt == 387, "[KG-ADHERENCE] KG: ’UTM_SOURCE NULL -> direct: 387/5053 sessions’.", ) def test_utm_medium_null_anchor_402(conn): c, db_type = conn cnt = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS) + " WHERE utm_medium IS NULL", ) require( cnt == 402, "[KG-ADHERENCE] KG: ’UTM_MEDIUM NULL -> none: 402/5053 sessions’.", ) def test_utm_campaign_null_anchor_3663(conn): c, db_type = conn cnt = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS) + " WHERE utm_campaign IS NULL", ) require( cnt == 3663, "[KG-ADHERENCE] KG: ’UTM_CAMPAIGN NULL -> none: 3663/5053 sessions’.", ) # -- Output model: existence, columns, grain, uniqueness ------------- -------- def test_model_exists_as_table(conn): c, db_type = conn schema = _out_schema(db_type) if db_type == "duckdb": rows = execute_query( c, db_type, "SELECT table_name FROM information_schema.tables " "WHERE lower(table_schema)=lower(’" + schema + "’) " "AND lower(table_name)=lower(’" + MODEL + "’)", ) else: rows = execute_query( c, db_type, "SELECT table_name FROM information_schema.tables " "WHERE lower(table_schema)=lower(’" + schema + "’) " "AND lower(table_name)=lower(’" + MODEL + "’)", ) require( len(rows) >= 1, "[KG-ADHERENCE] KG: Required output dbt model ’marts/marketing /" "rpt_attribution_fixed.sql’ materializing to schema ’" + schema + "’.", ) def test_model_has_grain_columns(conn): 56 c, db_type = conn schema = _out_schema(db_type) cols = _columns(c, db_type, schema, MODEL) missing = [g for g in GRAIN_COLS if g not in cols] require( len(missing) == 0, "[KG-ADHERENCE] KG grain: ’one row per (attribution_date, channel, medium, " "campaign)’. Missing grain columns: " + str(missing), ) def test_model_has_metric_columns(conn): c, db_type = conn schema = _out_schema(db_type) cols = _columns(c, db_type, schema, MODEL) needed = ["sessions", "conversions", "conversion_rate"] missing = [n for n in needed if n not in cols] require( len(missing) == 0, "[KG-ADHERENCE] KG metrics: sessions, conversions, conversion_rate " "(= conversions / sessions). Missing: " + str(missing), ) def test_model_has_revenue_column(conn): c, db_type = conn schema = _out_schema(db_type) cols = _columns(c, db_type, schema, MODEL) require( "attributed_revenue" in cols, "[KG-ADHERENCE] KG: attributed_revenue derived from t2_event_value else " "stg_ga__events.EVENT_VALUE, summed per session first.", ) def test_model_grain_is_unique(conn): c, db_type = conn schema = _out_schema(db_type) dup = execute_query( c, db_type, "SELECT attribution_date, channel, medium, campaign, COUNT( * ) AS n " "FROM " + _fqn(schema, MODEL) + " GROUP BY attribution_date, channel, medium, campaign HAVING COUNT( * ) > 1", ) require( len(dup) == 0, "[KG-ADHERENCE] KG grain: ’one row per (attribution_date, channel, medium, " "campaign)’ | the grain key must be unique (no duplicate combinations).", ) def test_model_row_count_positive(conn): c, db_type = conn schema = _out_schema(db_type) cnt = execute_scalar( c, db_type, 57 "SELECT COUNT( * ) FROM " + _fqn(schema, MODEL), ) require( cnt is not None and cnt > 0, "[UNIVERSAL] Output model must produce at least one grain row .", ) # -- Data quality invariants (Output requirements) ------------------- -------- def test_no_null_required_columns(conn): c, db_type = conn schema = _out_schema(db_type) rows = execute_query( c, db_type, "SELECT * FROM " + _fqn(schema, MODEL) + " WHERE attribution_date IS NULL OR channel IS NULL OR medium IS NULL " "OR campaign IS NULL OR sessions IS NULL OR conversions IS NULL " "OR conversion_rate IS NULL", ) require( len(rows) == 0, "[UNIVERSAL] Output requirements forbid NULL in required columns " "(grain keys, sessions, conversions, conversion_rate).", ) def test_conversion_rate_in_range(conn): c, db_type = conn schema = _out_schema(db_type) rows = execute_query( c, db_type, "SELECT conversion_rate FROM " + _fqn(schema, MODEL) + " WHERE conversion_rate < 0.0 OR conversion_rate > 1.0", ) require( len(rows) == 0, "[UNIVERSAL] KG: ’conversion_rate = conversions / sessions’ must be in " "[0.0, 1.0].", ) def test_conversions_le_sessions(conn): c, db_type = conn schema = _out_schema(db_type) rows = execute_query( c, db_type, "SELECT sessions, conversions FROM " + _fqn(schema, MODEL) + " WHERE conversions > sessions", ) require( len(rows) == 0, "[KG-ADHERENCE] KG: converted sessions are a subset of all sessions | " "conversions must never exceed sessions.", ) 58 def test_sessions_nonnegative(conn): c, db_type = conn schema = _out_schema(db_type) rows = execute_query( c, db_type, "SELECT sessions FROM " + _fqn(schema, MODEL) + " WHERE sessions IS NULL OR sessions < 0", ) require( len(rows) == 0, "[UNIVERSAL] sessions must be a non-negative count of distinct sessions.", ) def test_conversions_nonnegative(conn): c, db_type = conn schema = _out_schema(db_type) rows = execute_query( c, db_type, "SELECT conversions FROM " + _fqn(schema, MODEL) + " WHERE conversions IS NULL OR conversions < 0", ) require( len(rows) == 0, "[UNIVERSAL] conversions must be a non-negative count.", ) # -- conversion_rate formula recomputation --------------------------- --------- def test_conversion_rate_formula(conn): c, db_type = conn schema = _out_schema(db_type) # divide-by-zero guard: rate must be 0 (or NULL-guarded) when sessions=0. rows = execute_query( c, db_type, "SELECT sessions, conversions, conversion_rate FROM " + _fqn( schema, MODEL) + " WHERE sessions > 0 AND " "ABS(conversion_rate - ROUND(CAST(conversions AS DOUBLE)/ sessions, 4)) > 0.0001", ) require( len(rows) == 0, "[KG-ADHERENCE] KG: ’conversion_rate = conversions / sessions ... rounded " "to 4 decimals’ with divide-by-zero handled.", ) def test_conversion_rate_zero_sessions_guard(conn): c, db_type = conn schema = _out_schema(db_type) rows = execute_query( c, db_type, "SELECT * FROM " + _fqn(schema, MODEL) + " WHERE sessions = 0 AND conversion_rate IS NULL", ) require( len(rows) == 0, 59 "[KG-ADHERENCE] KG: ’divide-by-zero must be handled (NULLIF- style guard)’ | " "conversion_rate should not be NULL/undefined when sessions = 0.", ) # -- Cross-model reconciliation against confirmed sources ------------ --------- def test_total_sessions_reconciles_to_90day_window(conn): c, db_type = conn schema = _out_schema(db_type) # KG: 90-day trailing window anchored to MAX(CAST(session_start AS DATE)) # from stg_ga__sessions itself, counting distinct session_id. src_total = execute_scalar( c, db_type, "WITH anchored AS (" " SELECT session_id, CAST(session_start AS DATE) AS sd," " (SELECT MAX(CAST(session_start AS DATE)) FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS) + ") AS max_sd" " FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS) + ") SELECT COUNT(DISTINCT session_id) FROM anchored" " WHERE sd > max_sd - 90", ) model_total = execute_scalar( c, db_type, "SELECT SUM(sessions) FROM " + _fqn(schema, MODEL), ) require( src_total is not None and model_total is not None and abs(int(model_total) - int(src_total)) <= 0, "[UNIVERSAL] KG: ’90-day trailing window anchored to " "MAX(CAST(session_start AS DATE))’ with distinct session counting | " "SUM(sessions) in the mart must equal distinct sessions in that window " "(model=" + str(model_total) + ", source=" + str(src_total) + ").", ) def test_attribution_date_within_90day_window(conn): c, db_type = conn schema = _out_schema(db_type) max_sd = execute_scalar( c, db_type, "SELECT MAX(CAST(session_start AS DATE)) FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS), ) rows = execute_query( c, db_type, "SELECT attribution_date FROM " + _fqn(schema, MODEL) + " WHERE CAST(attribution_date AS DATE) <= " "(SELECT MAX(CAST(session_start AS DATE)) - 90 FROM " + _fqn(SRC_SCHEMA, STG_SESSIONS) + ")", ) require( len(rows) == 0, "[KG-ADHERENCE] KG: 90-day trailing window anchored to " "MAX(CAST(session_start AS DATE)) (max observed 2026-01-03). " "All attribution_date values must fall inside that window ( max_sd=" + 60 str(max_sd) + ").", ) def test_channel_defaults_direct_present_or_valid(conn): c, db_type = conn schema = _out_schema(db_type) # KG: UTM_SOURCE NULL -> ’direct’ default. No NULL/empty channel should remain. rows = execute_query( c, db_type, "SELECT channel FROM " + _fqn(schema, MODEL) + " WHERE channel IS NULL", ) require( len(rows) == 0, "[KG-ADHERENCE] KG: ’UTM_SOURCE NULL -> direct’ | channel must never be " "NULL in the output.", ) def test_medium_defaults_none_present_or_valid(conn): c, db_type = conn schema = _out_schema(db_type) rows = execute_query( c, db_type, "SELECT medium FROM " + _fqn(schema, MODEL) + " WHERE medium IS NULL", ) require( len(rows) == 0, "[KG-ADHERENCE] KG: ’UTM_MEDIUM NULL -> none’ | medium must never be NULL.", ) def test_campaign_defaults_none_present_or_valid(conn): c, db_type = conn schema = _out_schema(db_type) rows = execute_query( c, db_type, "SELECT campaign FROM " + _fqn(schema, MODEL) + " WHERE campaign IS NULL", ) require( len(rows) == 0, "[KG-ADHERENCE] KG: ’UTM_CAMPAIGN NULL -> none’ | campaign must never be " "NULL.", ) # -- Revenue: dedup-before-aggregate & non-negativity ---------------- --------- def test_attributed_revenue_nonnegative(conn): c, db_type = conn schema = _out_schema(db_type) rows = execute_query( c, db_type, "SELECT attributed_revenue FROM " + _fqn(schema, MODEL) + " WHERE attributed_revenue < 0", ) 61 require( len(rows) == 0, "[UNIVERSAL] attributed_revenue must be non-negative.", ) def test_revenue_not_inflated_by_fanout(conn): c, db_type = conn schema = _out_schema(db_type) # KG: sum revenue per session first, then aggregate | to avoid 1: ̃3.1 fanout # inflation. Total attributed_revenue must not exceed the total per-session # summed EVENT_VALUE upper bound across the whole events table. total_model = execute_scalar( c, db_type, "SELECT COALESCE(SUM(attributed_revenue),0) FROM " + _fqn( schema, MODEL), ) # Per-session summed event value (numeric cast), a safe upper bound. if db_type == "duckdb": cast_expr = "TRY_CAST(event_value AS DOUBLE)" else: cast_expr = "TRY_CAST(event_value AS DOUBLE)" upper = execute_scalar( c, db_type, "WITH per_session AS (" " SELECT session_id, SUM(" + cast_expr + ") AS rev" " FROM " + _fqn(SRC_SCHEMA, STG_EVENTS) + " WHERE session_id IS NOT NULL GROUP BY session_id" ") SELECT COALESCE(SUM(rev),0) FROM per_session", ) require( total_model is not None and upper is not None and float(total_model) <= float(upper) + 0.01, "[KG-ADHERENCE] KG: ’sum revenue per session first, then aggregate ... to " "avoid inflation from the confirmed 1: ̃3.1 session->event-row fanout’. " "Total attributed_revenue (" + str(total_model) + ") must not exceed the " "per-session summed EVENT_VALUE bound (" + str(upper) + ").", ) def test_revenue_zero_when_purchase_absent(conn): c, db_type = conn schema = _out_schema(db_type) # KG structural risk: EVENT_NAME=’purchase’ has 0 matches under any casing. # A spec-faithful (literal ’purchase’) solution yields attributed_revenue = 0 # for every row. If the agent used a substitute event token, this test flags # it so the choice is deliberate & justified rather than accidental. purchase_events = execute_scalar( c, db_type, "SELECT COUNT( * ) FROM " + _fqn(SRC_SCHEMA, STG_EVENTS) + " WHERE event_name = ’purchase’", ) total_model = execute_scalar( 62 c, db_type, "SELECT COALESCE(SUM(attributed_revenue),0) FROM " + _fqn( schema, MODEL), ) if purchase_events == 0: require( float(total_model) == 0.0, "[KG-ADHERENCE] KG structural risk: literal EVENT_NAME=’ purchase’ -> " "0 rows, so under the spec’s purchase rule attributed_revenue is 0 for " "every output row. Got total=" + str(total_model) + " | if non-zero, " "the agent substituted a different event token (must be justified).", ) else: require(True, "") # -- Metric consistency: conversions must reflect converted sessions - -------- def test_conversions_consistent_with_truthy_bound(conn): c, db_type = conn schema = _out_schema(db_type) # KG: converted numerator depends on is_converted interpretation (47 literal # or 160 truthy). Total conversions in the mart must not exceed the truthy # upper bound of 160 (within the 90-day window it is <= 160). total_conv = execute_scalar( c, db_type, "SELECT COALESCE(SUM(conversions),0) FROM " + _fqn(schema, MODEL), ) require( total_conv is not None and int(total_conv) <= 160, "[KG-ADHERENCE] KG is_converted ambiguity: truthy set (’1’,’Y ’,’true’) = " "160 sessions is the maximum eligible. SUM(conversions)=" + str(total_conv) + " must not exceed 160.", ) IMATH & SYSTEM BENCHMARK OPTIMIZATION I.1DETAILED METHODOLOGY All math and ADRS runs share a 100-iteration budget and a matched backbone (claude-sonnet-5 or openai-gpt-5.6-terra). AdaEvolve, OpenEvolve, and GEPA mutate through a stateless completion; CoCoEvolve uses the agentic loop of Appendix C.1, skipping the InfoGain branch because these tasks have no instance pool. The AdaEvolve row in Table 4 reruns AdaEvolve’s control loop with the same backbone model as CoCoEvolve. Matching the backbone removes model choice as an explanation for the observed gap between these search setups. We report the best objective reached within the shared budget. Task instances and metrics. Table 17 records the instance size, the scalar reported in the main ta- bles, and the Human/SOTA source. Higher is better except Cloudcast, which reports total broadcast cost. 63 TaskInstanceReported metricHuman/SOTA Circle Packing (Square) N =26, unit squaresum of radii2.63598 Circle Packing (Rect) N =21, perimeter 4sum of radii2.36583 Heilbronn (Triangle) N =11min triangle area / host area0.0365 Heilbronn (Convex)N =13min triangle area / hull area0.0309 MinMax Distanced=3, N =14(min/ max) 2 0.24004 Signal Processingcausal window filter overall score composite0.718 Cloudcast (↓)multi-cloud broadcasttotal cost626.2 EPLBMoE expert placementbalancedness/speed mean0.1265 PrismGPU model placement1/avg kvpr + success21.89 LLM-SQLcolumn reorder0.95·hit-rate +0.05·runtime0.692 TXNschedulingcombined throughput-style objective2,725 Table 17: Math and ADRS evaluator settings. Math Human/SOTA is the better of the published human reference and the best previously published automated result, from AlphaEvolve (Novikov et al., 2025) and AdaEvolve (Cemri et al., 2025). Systems Human/SOTA values are the single AdaEvolve references. I.2HIGHER-PRECISION MATH AND ADRS TABLES The main-text table rounds values for readability. Tables 18 and 19 report values at greater precision and retain rankings computed from the underlying values. Circle Packing (Square)Circle Packing (Rect)Heilbronn (Triangle)Heilbronn (Convex)MinMax DistanceSignal Processing StrategyAvgBestAvgBestAvgBestAvgBestAvgBestAvgBest Human/SOTA–2.63598–2.36583–0.0365–0.0309–0.24004–0.718 Backbone: Sonnet-5 OpenEvolve2.63598± 0.000002.635982.36557± 0.000452.365830.03197± 0.003170.03460.02775± 0.002160.029000.23780± 0.003090.239910.6499± 0.03960.6727 GEPA2.60218± 0.058562.635982.36530± 0.000912.365830.03588± 0.000800.03650.03018± 0.001320.030940.24005± 0.000000.240050.6031± 0.04810.6309 AdaEvolve2.63329± 0.002582.635982.36423± 0.002502.365830.03538± 0.001960.03650.02913± 0.001480.030760.24005± 0.000000.240050.7470± 0.02470.7656 CoCoEvolve2.63542± 0.000982.635982.36285± 0.005162.365830.03529± 0.001850.03650.03066± 0.000360.030940.24005± 0.000000.240050.7656± 0.02430.7841 Backbone: GPT-5.6-terra OpenEvolve2.63598± 0.000002.635982.36478± 0.000912.365830.03038± 0.000180.03050.02483± 0.002260.027090.23974± 0.000160.239830.5653± 0.03800.6075 GEPA2.63435± 0.002832.635982.36407± 0.001862.365830.03507± 0.000240.035210.02607± 0.003010.027810.24005± 0.000000.240050.6327± 0.04910.6610 AdaEvolve2.63315± 0.002002.634312.36584± 0.000012.365840.03457± 0.000590.03520.02353± 0.001040.024130.23982± 0.000190.240020.7921± 0.01970.8133 CoCoEvolve2.63375± 0.002552.635982.36378± 0.003542.365830.03408± 0.002140.03650.02544± 0.002050.027120.24005± 0.000000.240050.7563± 0.06080.80478 Table 18: Mathematical optimization benchmarks. We report Mean± Std and Best objective val- ues. Higher is better for all metrics. (N values: Square=26, Rect=21, Triangle=11, Convex=13; MinMax d=3, N =14). See Table 17. Human/SOTA is the better of the published Human ref- erence and the best previously published automated result on each problem, drawn from AlphaE- volve Novikov et al. (2025) and AdaEvolve Cemri et al. (2025). Values at greater precision corre- sponding to the main-paper Table 4. Bold/underline follow the same within-backbone ranking on the underlying values. Cloudcast (↓)EPLBPrismLLM-SQLTXN StrategyAvgBestAvgBestAvgBestAvgBestAvgBest Human/SOTA–626.2–0.1265–21.89–0.692–2,725 Backbone: Sonnet-5 OpenEvolve660.693089± 0.007070660.6890070.1273± 0.00010.127326.236± 0.03226.2560.7132± 0.01150.72633,945± 1234,016 GEPA621.090169± 5.213452618.0801810.1332± 0.01030.145126.256± 0.00026.2560.5235± 0.37330.75223,874± 2184,000 AdaEvolve618.096950± 0.008222618.0876000.1446± 0.00060.145026.391± 0.23326.6600.7235± 0.01610.73284,115± 174,132 CoCoEvolve618.063457± 0.006616618.0596370.1454± 0.00020.145526.256± 0.00026.2560.7280± 0.00580.73164,292± 554,348 Backbone: GPT-5.6-terra OpenEvolve618.096205± 0.005931618.0927810.1272± 0.00000.127226.076± 0.13626.1520.7242± 0.01030.73014,107± 1344,184 GEPA618.080973± 0.019122618.0699330.1272± 0.00000.127226.259± 0.00526.2640.7193± 0.00760.72374,268± 744,348 AdaEvolve618.085178± 0.013202618.0699330.1415± 0.00600.145425.674± 0.08025.7340.7168± 0.00650.72364,144± 204,167 CoCoEvolve618.081749± 0.010282618.0710970.14472± 0.000350.145126.338± 0.14126.5010.7283± 0.00200.73024,250± 814,329 Table 19: ADRS systems benchmarks. We report Mean± Std and Best objective values. Higher is better, except for Cloudcast where lower (↓) is better. Instance sizes and metrics are in Table 17. Human/SOTA values are from Cemri et al. (2025), which reports only a single reference value. Values at greater precision corresponding to the main-paper Table 4. Bold/underline follow the same within-backbone ranking on the underlying values. Signal Processing’s overall score is a weighted blend of a composite J (θ) (slope changes, lag, error, false reversals), smoothness, correlation, noise reduction, and success rate. 64 Signal Processing strategy convergence. Surprisingly, AdaEvolve and CoCoEvolve indepen- dently converge to essentially the same champion strategy: both fit the complete signal with an L1 data term and an L1 second-difference penalty, expressed as the same sparse linear program, and return the aligned tail of the resulting piecewise-linear trend. Their material difference is only how the regularization weight is chosen. CoCoEvolve scales a fixed coefficient by a robust noise estimate, whereas AdaEvolve uses a clipped noise-to-motion rule. In the reported exchange experi- ment, swapping the regularization rules also swaps the measured performance. The two champions therefore share the same broad filtering approach, and the regularization rule appears to explain their score difference in this comparison. I.3PER-TASK ANALYSIS Across both backbones, CoCoEvolve is best or tied best in 13 of 24 math cells and 12 of 20 ADRS cells when Avg and Best are counted separately. It leads 25 of 44 cells overall, compared with 18 for GEPA, 13 for AdaEvolve, and 6 for OpenEvolve. On Sonnet-5, CoCoEvolve has the highest Heilbronn Convex mean and ties GEPA on Best. In ADRS, it leads both Avg and Best on Cloudcast, EPLB, and TXN, and it leads LLM-SQL on Avg. AdaEvolve leads Prism, while GEPA leads LLM-SQL on Best. On GPT-5.6-terra, the leaders are distributed across all four methods. CoCoEvolve leads Prism and LLM-SQL and has the highest Heilbronn Triangle Best. AdaEvolve leads Signal Processing and Circle Packing (Rect). GEPA leads Heilbronn Convex and TXN and has the best Cloudcast Avg. OpenEvolve has the best Circle Packing (Square) mean. Circle Packing at both instances and MinMax Distance produce several ties at or near the reference values, so their means rather than their Bests are the informative column. The traces below examine two patterns in the table. Heilbronn Convex tests whether stalled trials recover near a known reference score; TXN shows a structural rewrite with room for a larger gain. Circle Packing provides a saturated case where final best scores are tied, and Cloudcast shows several methods finishing close together. These traces explain selected runs, not every aggregate result. InfoGain is inactive here: these tasks have no instance pool. Figures. Heilbronn Convex layouts and best-so-far: Figures 4–5 (Appendix I.3.1). Circle Packing (Square) layouts and best-so-far: Figures 6–7 (Appendix I.3.2). I.3.1HEILBRONN CONVEX (N =13) Setup. The objective is the ratio of the smallest triangle area among N =13 points to the area of their convex hull; Human/SOTA is 0.0309. The seed returns 13 uniform random points (0.00032). All four methods run the matched 100-iteration Sonnet-5 protocol of Appendix I.1; CoCoEvolve’s Avg in Table 4 is three retained trials. GEPA ties CoCoEvolve on Best, so the analysis below is not a unique-champion story: it is why Avg stays high when two of those three trials start badly. Figure 4 shows each method’s Best-cell geometry. Figure 5 overlays one Best-cell trace per method and a second retained CoCoEvolve trial (dotted) that recovers from a stall; both CoCoEvolve traces enter the published Avg. Search. The three retained CoCoEvolve finals are 0.03094, 0.03078, and 0.03025 (Table 4: 0.03066± 0.00036). Two of the three spend most of the budget near 0.028 and still finish at the ceiling, which is what keeps Avg above GEPA and the standard deviation a quarter of GEPA’s. The solid trace is the trial that does not need a recovery. Iterations 1–2 replace the random seed with from-scratch solvers (0.0182, then 0.0269). The reflector then keeps a constrained-NLP fam- ily (scipy.optimize.minimize, SLSQP) and a hillclimb of that family lands Human/SOTA at iteration 9, with a scratch self-check before each commit. GEPA reaches the same 0.03094 at iteration 39. That is sample efficiency on one trial, not the claim for the column. The dotted trial is the CMA-ES recovery and contributes to the three-run Avg. Its first rewrite scores 0.0234 and stalls near 0.0281 from iteration 31 while the mutator retries SciPy global solvers. At iteration 75, the reflector marks that family as exhausted and proposes a NumPy CMA-ES. A scratch prototype reaches 0.03035 at iteration 78. At iteration 89, the mutator returns to a dual-annealing 65 Seed (0.00032) OpenEvolve (0.02900) GEPA (0.03094) AdaEvolve (0.03076) CoCoEvolve (0.03094) Figure 4: Heilbronn Convex (N =13) on Sonnet-5. Point sets in the unit square: hull vertices (navy), interior points (lighter), hull outlined, smallest triangle in red. Seed is unstructured. OpenEvolve reaches a weaker 13-gon (0.02900); GEPA, AdaEvolve, and CoCoEvolve all reach structured 13- gons at their Best-cell scores. 0255075100 Iteration 0.018 0.020 0.022 0.024 0.026 0.028 0.030 0.032 Best min-triangle / hull SOTA @ 9 CoCoEvolve 0.03094 CoCoEvolve 0.03078 GEPA 0.03094 AdaEvolve 0.03076 OpenEvolve 0.02900 Human/SOTA 0.0309 Figure 5: Heilbronn Convex best-so-far on Sonnet-5, cropped just below the first rewrite. Solid traces are each method’s Best-cell run; Human/SOTA is the dashed green line. The dotted navy trace is a second retained CoCoEvolve trial: it stalls near 0.0281 and recovers to 0.03078. Solid CoCoEvolve reaches Human/SOTA at iteration 9. The claim is Avg, not the tied Best. program from iteration 19, which scored 0.02648, rather than using the current best CMA-ES child. It queries the ranked archive and adds CMA-ES with an exact-bilinear SLP polish, producing the 0.03078 champion. The third retained trial (not plotted) is the same mechanism with a different family. It crawls from 0.0244 to 0.0275 over 73 iterations of scipy global solvers, then implements Tutte barycentric em- bedding (scipy.sparse.linalg.spsolve) at iteration 86 and jumps to 0.03025. Either recovery is load-bearing for the Avg win. If the CMA-ES trial had remained at 0.0281 and the Tutte trial at its pre-jump 0.0275, the three-run mean would be 0.0289, below GEPA. If only one of the two recovered, the mean would still sit near 0.0297. Heilbronn therefore shows how historical synthesis improves reliability near a ceiling. The ledger triggers a new family after a stall, and the archive query transfers that family to a viable island. This mechanism explains the high Avg rather than a uniquely high Best. I.3.2CIRCLE PACKING (SQUARE) Setup. The objective is the sum of radii of N =26 non-overlapping circles in the unit square; Human/SOTA is 2.63598. The seed is a two-ring heuristic (0.96). This instance saturates: every method in Table 4 reaches Human/SOTA in at least one Sonnet-5 trial, so Best cannot rank them. The traces below are therefore a budget-use comparison, not a final-cell comparison. Figure 6 shows 66 one ceiling packing per method. Figure 7 overlays the Best-cell best-so-far of each method, cropped near the ceiling, under the same 100-iteration window. 0.00.51.0 Seed (0.96) 0.0 0.5 1.0 01 2 3 4 5 6 7 8 0.00.51.0 OpenEvolve (2.63598) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0.00.51.0 GEPA (2.63598) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0.00.51.0 AdaEvolve (2.63598) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0.00.51.0 CoCoEvolve (2.63598) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Figure 6: Circle Packing (Square, N =26) on Sonnet-5. Circles in the unit square, labeled 0–25. The seed is a two-ring heuristic (0.96); OpenEvolve, GEPA, AdaEvolve, and CoCoEvolve are size- biased ceiling packings at Human/SOTA (2.63598). 0255075100 Iteration 2.60 2.61 2.62 2.63 2.64 2.65 Best sum of radii GEPA SOTA @ 8 CoCo @ 24 CoCoEvolve 2.63598 GEPA 2.63598 AdaEvolve 2.63598 OpenEvolve 2.6298 @ 100 Human/SOTA 2.63598 Figure 7: Circle Packing (Square, N =26) best-so-far on Sonnet-5, cropped near the ceiling. Solid traces are each method’s Best-cell run; Human/SOTA is the dashed green line. GEPA first reaches 2.63598 at iteration 8, CoCoEvolve at iteration 24, AdaEvolve at iteration 91. OpenEvolve is at 2.6298 at iteration 100 and only hits the ceiling at iteration 111, outside the shared window. Search. This is a saturation control, not a CoCoEvolve win. Once every method can hit the ceiling, the remaining signal is how the shared budget is spent getting there. On the traces in Figure 7, GEPA is first (iteration 8), CoCoEvolve is second (iteration 24), AdaEvolve reaches the ceiling at iteration 91, and OpenEvolve is still at 2.6298 at iteration 100 (its Best of 2.63598 is from iteration 111, outside the shared window). We do not read first-to-ceiling as a method ranking: GEPA winning that race on a saturated packing instance is compatible with Heilbronn, where the same protocol separates the methods on Avg rather than Best. I.3.3CLOUDCAST Setup. The objective is the simulator’s true union-of-edges cost of a multi-cloud broadcast tree (lower is better); Human/SOTA is 626.2. The seed enumerates Azure single- and two-hub plans ranked by a node-count/bottleneck proxy, with an arborescence fallback, and scores 974. The analysis below is the CoCoEvolve Best-cell Sonnet-5 run under the same 100-iteration protocol. OpenEvolve remains far from the others; GEPA, AdaEvolve, and CoCoEvolve all finish in a tight 618 cluster (Table 4), so this subsection is about the construction that reached that cluster, not about ranking inside it. 67 Search. The champion no longer searches only for hubs. It evaluates six structurally distinct tree families: terminal MSA, relay hub, capacitated min-cost flow, Dreyfus–Wagner DP, resistance rank- ing, and TSP relay. A proxy replays the simulator’s cost model and selects the cheapest candidate. The first two iterations reduce cost from 974 to 636 by reading the cost graph and instance config- uration and by testing scratch candidates against the live evaluator. Later cross-lineage grafts add Dreyfus–Wagner ranking and a TSP-relay construction, which produce the remaining improvement. The trace documents a substantial rewrite of the seed’s search procedure. Because GEPA, AdaE- volve, and CoCoEvolve finish close together, Cloudcast gives little evidence for ranking their final scores. TXN provides a clearer example of a large score change after a structural rewrite. I.3.4TXN SCHEDULING Setup. The objective is a combined throughput-style score of a lock-aware transaction schedule (higher is better); Human/SOTA is 2,725. The seed is tabu search over swap and or-opt-1 from Copeland / MDS / assignment-problem / greedy starts, and scores 2,725. The reported runs show substantial headroom above that seed and reference value. Both the mean and best scores can there- fore distinguish methods here. The analysis below is the CoCoEvolve Best-cell Sonnet-5 run that produced 4,348, under the same 100-iteration protocol. Search. The champion replaces the seed’s tabu loop with a three-stage pipeline. An assignment- problem relaxation and a single-path greedy construction from the top Copeland candidate each build a starting schedule, and the cheaper one is kept; simulated annealing then applies double- bridge reheating, and first-improvement local search performs the final cleanup. Copeland ranking and an EDA over pairwise order probabilities first move the score above the seed. The EDA also queries the ranked archive instead of editing only the parent. The final program uses assignment/greedy construction followed by simulated annealing, not a retuned tabu tenure. TXN is therefore the far-from-ceiling counterpart to Heilbronn. In Heilbronn, the ledger improves reliability by rescuing stalled trials. In TXN, the same keep-or-switch loop replaces the seed meta- heuristic and improves both Avg and Best beyond the other methods. JPROMPT OPTIMIZATION J.1DETAILED METHODOLOGY We evaluate three tasks from GEPA’s benchmark suite. All three compare an unmodified seed, GEPA, and CoCoEvolve from the same initial program; each CoCoEvolve run uses 100 iterations. HotpotQA. The artifact is a four-stage retrieve-summarize-answer pipeline over a BM25 index of Wikipedia abstracts, with prompts for first-hop summarization (summarize1), second-hop query generation (create queryhop2), second-hop summarization (summarize2), and final an- swering (finalanswer). Each mutation changes one prompt while retrieval remains fixed, and GEPA’s round-robin selector mutates the same four components in the same cycle. We use GEPA’s official 150/300/300 train, validation, and held-out-test split. The GEPA search budgets were chosen to be broadly comparable to the 100-iteration CoCoEvolve runs, although the two methods account for evaluations differently. The frozen Sonnet-5 backbone serves as both mutator and student. GEPA uses the same Benchmark base class for AIME, HotpotQA, and IFBench. The artifact runner passes individual query instances in separate training and validation sets for all three tasks. GEPA samples three training examples by default for each reflective step. A proposed edit must strictly improve its parent’s score on that minibatch. Each accepted candidate then triggers a full validation-set evaluation, whose per-example scores inform the Pareto selector’s choice of future parents. AIME. The evolved artifact is a single chain-of-thought system prompt that a frozen student model reads with each competition math problem to produce an integer answer. GEPA’s fixed-seed split draws 45 training and 45 validation problems from AI-MO/aimo-validation-aime; its disjoint test set contains 30 problems from MathArena/aime 2025.AIME uses 68 openai-gpt-5.6-luna as both mutator and student. An initial Sonnet-5 run gave the seed a perfect training score, leaving no signal for optimization. IFBench. Theartifacthastwoprompt-drivenstages, generate responseand ensurecorrectresponse.The evaluator scores the fraction of each query’s instruc- tion constraints that the response satisfies. It supports 83 checker types: 25 from IFEval and 58 IFBench-native types, including word and letter counts, formatting, and keyword use. Eight leniency variants account for markdown emphasis and optional leading or trailing lines. Neither stage sees the constraint list. GEPA’s fixed split uses the first 300 rows of its training file for validation and samples 150 training rows from the next 300 with a fixed seed. We use its bundled 294-row test file unchanged; it has six fewer rows than the public IFBench release. IFBench also uses openai-gpt-5.6-luna as both mutator and student. The IFBench run uses the default FullEvaluationPolicy and 100 concurrent workers. Repeated evaluation. With openai-gpt-5.6-luna, we evaluated the same evolved AIME prompt twice on the 45 training problems: once during optimization and once afterward. The second evaluation got five fewer answers correct, even though the prompt had not changed. The student’s nonzero decoding temperature can make its answers vary between evaluations. Each optimizer ran once and produced one final candidate. We evaluated that unchanged candidate three times on each AIME and IFBench split and report the mean and standard deviation in Table 5. These repeats measure evaluation variability, not search variability. On AIME test, five baseline questions, eight GEPA questions, and six CoCoEvolve questions changed correctness across repeats, even though the baseline’s aggregate test score was 25/30 every time. J.2AN EFFICIENCY VARIANT: SIM-GRAPH TARGETING AND CASCADED EVALUATION ON HOTPOTQA The HotpotQA run of Appendix J.1 scores its entire 150-question training pool and 300-question validation pool on every one of its 100 iterations, unconditionally – 450 real model-backed ques- tions per iteration regardless of how informative any single one is. DABStep’s own evaluator (Ap- pendix C.2, Appendix C.3) instead scores only a bounded, informativeness-targeted subset per it- eration and extrapolates the rest from the parent’s own history. We test a HotpotQA variant that combines similarity-based target selection, P0/P1/P2 cascaded evaluation, and mutator self-testing. The comparison measures this combined configuration. It cannot attribute any score or time differ- ence to one of the three changes alone. Setup. We keep the seed, 100-iteration budget, and backbone from Appendix J.1. We add three mechanisms. Target selection. A similarity graph and InfoGain selector operate over the 150-question training pool (Appendix C.3). Each mutation receives one selected focus question instead of the full undif- ferentiated pool. Cascaded evaluation. The P0/P1/P2 cascade mirrors the DABStep evaluator (Appendix F.1). P0 evaluates only the selected target and requires a score of 0.6. If P0 passes, P1 evaluates a stratified sample of 20 regression questions, or about 13% of the pool. One third of this sample is selected by similarity-weighted risk to the target, and two thirds are sampled uniformly from the parent’s tested history. P1 requires at least 0.9× the parent’s score on the same sample. If P1 passes, P2 explores similarity-ranked frontier tiers of 10 questions each. One random scout represents each tier, and the next tier opens only if the current tier scores at least 6/10. The training cascade evaluates at most⌊150/3⌋=50 questions per iteration, excluding the one-time seed bootstrap, mutator self-tests, and separately sampled validation passes. When the parent is already at ceiling on the target, a live pretest can bypass P0 and P1 and proceed directly to P2. The child’s score combines its updated per-question record with the parent’s inherited record and divides the total by 150. We do not use DABStep’s additive formula because that formula assumes sparse initial coverage. HotpotQA has dense coverage from the seed’s full-pool bootstrap, so the additive formula would double-count observations. 69 Mutator self-test. The mutator receives testdraft, with a budget of three calls per turn. It can score an in-progress edit directly on the selected focus question. The mutator prompt also instructs the model to verify its draft before finishing. Both features reproduce DABStep’s self-verification affordance (Appendix C.2). Validation is sampled independently with probability 0.1 per iteration. When validation is skipped, the child inherits the parent’s validation score and per-question record unchanged. EvaluatorTrainValTest Full pool, every iteration (Appendix J.1)0.8000.7830.793 Sim-graph target + P0/P1/P2 cascade0.8000.7900.767 Table 20: HotpotQA, Sonnet-5, CoCoEvolve: the published full-pool-every-iteration search versus the sim-graph-targeted/cascaded-evaluation variant of this appendix. Both rows are a single 100- iteration search, freshly re-scored against the identical train/val/test pools. Results. Across 100 iterations, the cascaded evaluator has a median wall-clock time of 121s per iteration. A full 450-question pass at the same concurrency takes roughly 900–1100s. The seed’s one-time full-pool bootstrap takes 1115s, which is consistent with that estimate. In the first 50 logged iterations, seven (14%) took 400–1100s; one was the seed bootstrap, and the others included a sampled pass over all 300 validation questions. Extrapolating that mix to 100 iterations gives an estimated 5.5–6 hours of evaluator time, versus an estimated 25–30 hours for full-pool evaluation at every iteration. The implied reduction is about 4–5×. These are estimates from the observed timing mix, not measured total run times. Analysis. The combined variant matches the full-pool run’s observed train accuracy (0.800) and has 0.007 higher validation accuracy, but its test accuracy is 0.026 lower, about eight questions out of 300. The timing estimate suggests lower evaluator cost. With one search and one final re-score per variant, the result does not establish equal held-out quality or isolate which added mechanism produced the difference. It also does not replace or improve the main HotpotQA result in Table 5. 70 KCOMPLETE BASE AND FINAL PROGRAMS The following unabridged listings pair the seed and final program from one run each of Heil- bronn Convex, Cloudcast, and TXN. The Cloudcast and TXN listings contain only the evolved function; module-level imports (e.g., networkx as nx, random) and harness classes such as BroadCastTopology are supplied by the surrounding benchmark file. K.1HEILBRONN CONVEX Heilbronn Convex: base program 1 import numpy as np 2 3 4 def heilbronn_convex13() -> np.ndarray: 5 """ 6 Construct an arrangement of n points on or inside a convex region in order to maximize the area of the 7 smallest triangle formed by these points. Here n = 13. 8 9 Returns: 10 points: np.ndarray of shape (13,2) with the x,y coordinates of the points. 11 """ 12 n = 13 13 rng = np.random.default_rng(seed=42) 14 points = rng.random((n, 2)) 15 return points Heilbronn Convex: final evolved program 1 """ 2 Heilbronn convex-13 placement via a from-scratch, pure-numpy CMA-ES 3 (Covariance Matrix Adaptation Evolution Strategy) driven directly on the 4 raw, non-smoothed maximin ratio (min triangle area / hull area) over all 5 26 raw coordinates, followed by an exact-bilinear trust-region Sequential 6 Linear Programming (SLP) local polish and a deterministic iterated local 7 search (ILS) refinement pass. 8 9 This replaces the previous strategy in this lineage (a single 10 ‘scipy.optimize.dual_annealing‘ call on the raw ratio, warm-started from a 11 regular 13-gon with a Powell local-search step) with a mechanistically 12 different two-stage pipeline: 13 14 Stage A (global, derivative-free): many independent from-scratch CMA-ES 15 restarts, each maintaining a persistent, adaptively-controlled global step 16 size (cumulative step-size adaptation) plus a covariance matrix blended 17 from a rank-one "remembered direction" term and a rank-mu "current elite 18 scatter" term. This is far more resistant to premature step-size collapse 19 on this rugged, non-smooth landscape than a single dual_annealing run’s 20 fixed cooling schedule, because sigma can re-expand if progress stalls 21 instead of only ever shrinking. 22 23 Stage B (local, exact-gradient): every one of the C(13,3)=286 triangle 24 areas is an EXACT bilinear (shoelace) function of the 26 coordinates, so 25 a trust-region Sequential Linear Programming (SLP) procedure can linearize 26 every triangle’s signed area around the current point, solve one small 27 epigraph LP (‘scipy.optimize.linprog‘, method="highs") for the best 28 bounded displacement, and accept/reject/resize the trust region via the 29 classical predicted-vs-actual ratio test against the TRUE (non-linearized) 30 ratio. This converges far closer to an exact local optimum of the maximin 31 problem than any further stochastic CMA-ES generations would, and is 32 distinct from the previous line’s derivative-free Powell polish (which 33 never exploits the exact bilinear structure of the triangle areas). 34 35 Stage C: a deterministic iterated local search (fixed, shrinking-noise- 36 scale perturbation kicks re-polished by the SLP procedure) around the best 37 SLP incumbent, to escape any shallow local optimum the trust-region 38 procedure settles into on its own. 39 40 Scaling vs. the smaller/faster variant of this same architecture previously 41 tried in this lineage (14 restarts x 500 generations, ̃25s total): this 42 version raises the restart count, per-restart generation budget, SLP outer- 43 iteration cap, and ILS iteration count substantially (to ̃80-150s total, 44 timed empirically via scratch runs of the identical CMA-ES/SLP code), since 45 the evaluator’s timeout (360s) leaves very large unused headroom relative to 46 both the previous ̃25s run and the ̃70s single dual_annealing call this 47 replaces. More restarts/generations of CMA-ES is the primary lever for 48 escaping local optima on this rugged landscape, so that budget increase is 49 prioritized over any other single change. Restart warm starts are also 50 diversified: half start from a jittered regular 13-gon (as before), the 51 other half from a jittered two-ring layout (an outer ring of points plus an 71 52 inner ring, rotated by a half-step), giving CMA-ES’s covariance adaptation a 53 second, structurally distinct family of basins to explore per restart batch. 54 55 Everything is fully deterministic (numpy ‘Generator‘ objects seeded with 56 fixed integers throughout, no unseeded randomness anywhere), so results are 57 exactly reproducible across runs and platforms. 58 """ 59 import numpy as np 60 from itertools import combinations 61 from scipy.spatial import ConvexHull 62 from scipy.optimize import linprog 63 64 _N = 13 65 _COMBOS = np.array(list(combinations(range(_N), 3))) # all C(13,3)=286 index triples 66 _TI, _TJ, _TK = _COMBOS[:, 0], _COMBOS[:, 1], _COMBOS[:, 2] 67 _NTRI = len(_TI) 68 _DIM = 2 * _N 69 70 71 def _hull_area(points: np.ndarray) -> float: 72 """Convex-hull area of a single (N,2) point set, or -1.0 if degenerate.""" 73 try: 74 ha = ConvexHull(points).volume 75 except Exception: 76 return -1.0 77 return ha if ha > 1e-12 else -1.0 78 79 80 def _ratio(points: np.ndarray) -> float: 81 """True (non-smooth) objective to maximize: (smallest triangle area) / (convex hull area).""" 82 p1, p2, p3 = points[_TI], points[_TJ], points[_TK] 83 cross = (p2[:, 0] - p1[:, 0]) * (p3[:, 1] - p1[:, 1]) - (p3[:, 0] - p1[:, 0]) * (p2[:, 1] - p1[:, 1]) 84 areas = 0.5 * np.abs(cross) 85 ha = _hull_area(points) 86 if ha <= 0.0: 87 return -1.0 88 return float(areas.min() / ha) 89 90 91 def _batch_scores(X: np.ndarray) -> np.ndarray: 92 """ 93 Vectorized raw-score evaluation of a whole population at once: every candidate’s 94 286 triangle areas are computed in one single broadcasted numpy expression across 95 the whole (popsize, 26) batch. The convex-hull area still needs one ‘ConvexHull‘ 96 call per candidate (qhull has no batched entry point), but that per-candidate loop 97 does none of the O(popsize * 286) triangle-area work, which is the part that 98 actually scales. 99 """ 100 popsize = X.shape[0] 101 pts = X.reshape(popsize, _N, 2) 102 p1, p2, p3 = pts[:, _TI, :], pts[:, _TJ, :], pts[:, _TK, :] 103 cross = (p2[:, :, 0] - p1[:, :, 0]) * (p3[:, :, 1] - p1[:, :, 1]) - ( 104 p3[:, :, 0] - p1[:, :, 0] 105 ) * (p2[:, :, 1] - p1[:, :, 1]) 106 min_areas = 0.5 * np.abs(cross).min(axis=1) 107 108 scores = np.full(popsize, -1.0) 109 for i in range(popsize): 110 ha = _hull_area(pts[i]) 111 if ha > 0.0: 112 scores[i] = min_areas[i] / ha 113 return scores 114 115 116 def _regular_13gon() -> np.ndarray: 117 """Defensive fallback: a plain regular 13-gon, always convex and non-degenerate.""" 118 angles = np.linspace(0, 2 * np.pi, _N, endpoint=False) 119 return np.stack([np.cos(angles), np.sin(angles)], axis=1) 120 121 122 def _jittered_13gon(rng: np.random.Generator, jitter: float = 0.05) -> np.ndarray: 123 """Regular 13-gon perturbed by small iid Gaussian jitter -- one CMA-ES restart init family.""" 124 base = _regular_13gon() 125 return (base + rng.normal(scale=jitter, size=base.shape)).reshape(-1) 126 127 128 def _jittered_two_ring(rng: np.random.Generator, m_outer: int = 8, jitter: float = 0.05) -> np.ndarray: 129 """ 130 Second, structurally distinct restart-init family: an outer ring of ‘m_outer‘ 131 points plus an inner ring of the remaining points (rotated by a half-step so no 132 point is radially aligned with an outer one), each perturbed by small iid 133 Gaussian jitter. Gives CMA-ES’s covariance adaptation a different starting basin 134 family than the single-ring regular 13-gon used by ‘_jittered_13gon‘. 135 """ 136 n_inner = _N - m_outer 137 angles_o = np.linspace(0, 2 * np.pi, m_outer, endpoint=False) 138 outer = np.stack([np.cos(angles_o), np.sin(angles_o)], axis=1) 139 apothem = np.cos(np.pi / m_outer) 140 r_inner = 0.45 * apothem 72 141 angles_i = np.linspace(0, 2 * np.pi, n_inner, endpoint=False) + (np.pi / m_outer) 142 inner = np.stack([r_inner * np.cos(angles_i), r_inner * np.sin(angles_i)], axis=1) 143 base = np.concatenate([outer, inner], axis=0) 144 return (base + rng.normal(scale=jitter, size=base.shape)).reshape(-1) 145 146 147 def _cma_es(mean0: np.ndarray, sigma0: float, popsize: int, n_gen: int, seed: int): 148 """ 149 A complete, from-scratch, pure-numpy (mu/mu_W, lambda)-CMA-ES loop (Hansen & 150 Ostermeier), maximizing ‘_batch_scores‘ over the flat 26-D coordinate vector. 151 152 Each generation: 153 1. Sample ‘popsize‘ candidates x = mean + sigma * B @ (D * z), z ̃ N(0, I) 154 (equivalently, x ̃ N(mean, sigmaˆ2 * C) with C = B @ diag(Dˆ2) @ B.T), 155 evaluated in one vectorized batch via ‘_batch_scores‘. 156 2. Rank-select the top ‘mu = popsize // 2‘ candidates by raw score and take 157 their log-rank-weighted mean as the new distribution mean. 158 3. Update the step-size evolution path p_sigma (cumulative step-size 159 adaptation, CSA) and rescale sigma so that p_sigma’s length tracks its 160 expected length under random selection -- sigma grows when steps are 161 longer/more aligned than expected (fast progress) and shrinks when they 162 are shorter (fine-tuning / stalling). 163 4. Update the covariance evolution path p_c, then blend the covariance 164 matrix with a rank-one term (outer(p_c, p_c): remembers the * direction * 165 of sustained recent progress) and a rank-mu term (the weighted outer 166 product of the current elites’ steps: captures the current generation’s 167 scatter). 168 5. Periodically re-eigendecompose C into (B, D) for the next generation’s 169 sampling and the Cˆ-1/2 needed by step 3. 170 A light-touch rescaling keeps the search numerically well-conditioned: since 171 ‘_ratio‘ is exactly scale-invariant, whenever the current mean’s own hull area 172 drifts far from 1 we rescale mean and sigma together by the same factor (a 173 pure similarity transform, so it changes nothing about the objective, only 174 keeps coordinate magnitudes in a numerically safe range across hundreds of 175 generations of unconstrained drift). 176 177 Returns (best_x, best_score): the single best-scoring 26-D vector and its raw 178 score ever observed across every generation of this run (not just the final 179 one, since CMA-ES’s stochastic sampling can visit its best point generations 180 before the distribution itself fully commits there). 181 """ 182 rng = np.random.default_rng(seed) 183 dim = _DIM 184 mu = popsize // 2 185 186 # Log-rank elite weights and the standard derived CMA-ES strategy parameters. 187 w_raw = np.log(mu + 0.5) - np.log(np.arange(1, mu + 1)) 188 weights = w_raw / w_raw.sum() 189 mu_eff = 1.0 / np.sum(weights ** 2) 190 191 c_sigma = (mu_eff + 2) / (dim + mu_eff + 5) 192 d_sigma = 1 + 2 * max(0.0, np.sqrt((mu_eff - 1) / (dim + 1)) - 1) + c_sigma 193 c = (4 + mu_eff / dim) / (dim + 4 + 2 * mu_eff / dim) 194 c1 = 2 / ((dim + 1.3) ** 2 + mu_eff) 195 cmu = min(1 - c1, 2 * (mu_eff - 2 + 1 / mu_eff) / ((dim + 2) ** 2 + mu_eff)) 196 chiN = np.sqrt(dim) * (1 - 1 / (4 * dim) + 1 / (21 * dim ** 2)) 197 198 mean = mean0.astype(float).copy() 199 sigma = float(sigma0) 200 C = np.eye(dim) 201 p_sigma = np.zeros(dim) 202 p_c = np.zeros(dim) 203 B = np.eye(dim) 204 D = np.ones(dim) 205 eig_gen = 0 206 207 best_score = -1e18 208 best_x = mean.copy() 209 210 for gen in range(n_gen): 211 Z = rng.standard_normal((popsize, dim)) 212 X = mean + sigma * (Z @ (B * D).T) 213 214 scores = _batch_scores(X) 215 order = np.argsort(-scores) 216 217 if scores[order[0]] > best_score: 218 best_score = float(scores[order[0]]) 219 best_x = X[order[0]].copy() 220 221 elite_X = X[order[:mu]] 222 mean_old = mean 223 mean = weights @ elite_X 224 225 # --- cumulative step-size adaptation (CSA) --- 226 y = (mean - mean_old) / sigma 227 c_inv_sqrt_y = B @ ((B.T @ y) / D) 228 p_sigma = (1 - c_sigma) * p_sigma + np.sqrt(c_sigma * (2 - c_sigma) * mu_eff) * c_inv_sqrt_y 229 sigma * = np.exp((c_sigma / d_sigma) * (np.linalg.norm(p_sigma) / chiN - 1)) 230 231 # --- covariance evolution path + rank-one/rank-mu covariance update --- 73 232 hs = 1.0 if ( 233 np.linalg.norm(p_sigma) / np.sqrt(1 - (1 - c_sigma) ** (2 * (gen + 1))) / chiN 234 ) < (1.4 + 2 / (dim + 1)) else 0.0 235 p_c = (1 - c) * p_c + hs * np.sqrt(c * (2 - c) * mu_eff) * y 236 237 elite_steps = (elite_X - mean_old) / sigma 238 rank_mu_term = (elite_steps.T * weights) @ elite_steps 239 C = ( 240 (1 - c1 - cmu) * C 241 + c1 * (np.outer(p_c, p_c) + (1 - hs) * c * (2 - c) * C) 242 + cmu * rank_mu_term 243 ) 244 245 # Numerical safety net: ‘_ratio‘ is exactly scale-invariant, so rescaling 246 # (mean, sigma) together by the same factor changes nothing about the 247 # objective but keeps coordinate magnitudes from drifting to extremes 248 # over hundreds of unconstrained generations. 249 mean_ha = _hull_area(mean.reshape(_N, 2)) 250 if mean_ha > 0.0 and (mean_ha < 0.1 or mean_ha > 10.0): 251 rescale = 1.0 / np.sqrt(mean_ha) 252 mean = mean * rescale 253 sigma * = rescale 254 255 # Periodic eigendecomposition refresh of C -> (B, D), amortized over 256 # roughly dim/(c1+cmu)/10 generations as in standard CMA-ES practice. 257 if gen - eig_gen > max(1, int(popsize / (c1 + cmu) / dim / 10)): 258 eig_gen = gen 259 C = np.triu(C) + np.triu(C, 1).T # enforce exact symmetry 260 eigvals, eigvecs = np.linalg.eigh(C) 261 eigvals = np.clip(eigvals, 1e-20, None) 262 D = np.sqrt(eigvals) 263 B = eigvecs 264 265 return best_x, best_score 266 267 268 # --------------------------------------------------------------------------- 269 # Trust-region Sequential Linear Programming (SLP) local polish. 270 # 271 # CMA-ES’s covariance-adapted Gaussian sampling is an excellent global/basin 272 # search, but being stochastic and derivative-free it never converges exactly 273 # onto a KKT point of the maximin-ratio problem -- its distribution keeps 274 # jittering around the optimum rather than landing on it. Every one of the 275 # C(13,3)=286 triangle areas is, however, an EXACT bilinear (shoelace) function 276 # of the 26 free coordinates: Area_ijk(x) = 0.5 * (xi * (yj-yk)+xj * (yk-yi)+xk * (yi-yj)). 277 # That lets us do far sharper local polishing than more CMA-ES generations 278 # could: fix every triangle’s current orientation sign s_ijk = sign(Area_ijk(x)) 279 # and solve the exact (to first order) trust-region epigraph LP 280 # maximize t s.t. s_ijk * (Area_ijk(x) + grad(Area_ijk)(x)ˆT d) >= t 281 # ||d||_inf <= r 282 # once per outer iteration via a single scipy.optimize.linprog(method="highs") 283 # call, then apply the classical trust-region accept/reject + grow/shrink test 284 # comparing the LP’s * predicted * improvement against the * actual * improvement 285 # of the true (non-linearized) ratio. 286 # --------------------------------------------------------------------------- 287 288 289 def _signed_areas_and_grad(x: np.ndarray): 290 """Exact signed area and gradient (w.r.t. the flat 26-D coordinate vector) of 291 every one of the 286 triangles, via the shoelace formula. Each triangle’s area 292 is a bilinear (degree-2) function of its three points’ six coordinates, so this 293 analytic gradient is exact -- the first-order Taylor expansion used by the SLP 294 step below is exact up to a purely quadratic remainder in the displacement.""" 295 pts = x.reshape(_N, 2) 296 xi, yi = pts[_TI, 0], pts[_TI, 1] 297 xj, yj = pts[_TJ, 0], pts[_TJ, 1] 298 xk, yk = pts[_TK, 0], pts[_TK, 1] 299 f = 0.5 * (xi * (yj - yk) + xj * (yk - yi) + xk * (yi - yj)) 300 grad = np.zeros((_NTRI, _DIM)) 301 rows = np.arange(_NTRI) 302 grad[rows, 2 * _TI] = 0.5 * (yj - yk) 303 grad[rows, 2 * _TI + 1] = 0.5 * (xk - xj) 304 grad[rows, 2 * _TJ] = 0.5 * (yk - yi) 305 grad[rows, 2 * _TJ + 1] = 0.5 * (xi - xk) 306 grad[rows, 2 * _TK] = 0.5 * (yi - yj) 307 grad[rows, 2 * _TK + 1] = 0.5 * (xj - xi) 308 return f, grad 309 310 311 def _slp_normalize(x: np.ndarray) -> np.ndarray: 312 """Recenter at the centroid and rescale so hull area is exactly 1 -- a pure 313 similarity transform that leaves the scale-invariant ‘_ratio‘ untouched but 314 keeps the LP’s coefficients well-conditioned across many outer iterations.""" 315 pts = x.reshape(_N, 2) 316 ha = _hull_area(pts) 317 if ha <= 1e-12: 318 ha = 1.0 319 pts = pts - pts.mean(axis=0) 320 pts = pts * np.sqrt(1.0 / ha) 321 return pts.reshape(-1) 322 74 323 324 def _slp_lp_step(x: np.ndarray, r: float): 325 """One trust-region LP subproblem: linearize every triangle’s signed area 326 around x and solve for the best epigraph displacement d with ||d||_inf <= r.""" 327 f, grad = _signed_areas_and_grad(x) 328 s = np.sign(f) 329 s[s == 0.0] = 1.0 330 sf = s * f 331 sgrad = grad * s[:, None] 332 A_ub = np.hstack([-sgrad, np.ones((_NTRI, 1))]) 333 b_ub = sf 334 c = np.zeros(_DIM + 1) 335 c[-1] = -1.0 336 bounds = [(-r, r)] * _DIM + [(None, None)] 337 try: 338 res = linprog(c, A_ub=A_ub, b_ub=b_ub, bounds=bounds, method="highs") 339 except Exception: 340 return None, None 341 if not res.success: 342 return None, None 343 return res.x[:_DIM], res.x[-1] 344 345 346 def _run_slp(x0: np.ndarray, max_outer: int = 200, r0: float = 0.2, 347 r_min: float = 1e-10, r_max: float = 0.5): 348 """Trust-region SLP / convex-concave procedure: repeatedly linearize, solve the 349 LP, and accept/reject/resize via the classical trust-region ratio test against 350 the true (non-linearized) ratio. Returns (best_x, best_ratio).""" 351 x = _slp_normalize(x0.copy()) 352 cur_ratio = _ratio(x.reshape(_N, 2)) 353 r = r0 354 for _ in range(max_outer): 355 if r < r_min: 356 break 357 d, t_pred = _slp_lp_step(x, r) 358 if d is None: 359 r * = 0.5 360 continue 361 x_new = x + d 362 new_ratio = _ratio(x_new.reshape(_N, 2)) 363 predicted_improve = t_pred - cur_ratio 364 if predicted_improve <= 1e-14: 365 r * = 0.5 366 continue 367 rho = (new_ratio - cur_ratio) / predicted_improve 368 if rho > 0.1: 369 x = _slp_normalize(x_new) 370 cur_ratio = new_ratio 371 if rho > 0.75: 372 r = min(r * 1.5, r_max) 373 else: 374 r * = 0.5 375 return x, cur_ratio 376 377 378 def heilbronn_convex13() -> np.ndarray: 379 """ 380 Construct an arrangement of n=13 points on or inside a convex region to maximize the 381 area of the smallest triangle formed by any three of them, normalized by the area of 382 their own convex hull. 383 384 Approach (multi-restart, from-scratch pure-numpy CMA-ES on the raw ratio, followed by 385 an exact-bilinear trust-region SLP local-polish phase and a deterministic ILS pass): 386 1. Run ‘_N_RESTARTS‘ independent CMA-ES searches (see ‘_cma_es‘), alternating between 387 two structurally distinct deterministic warm-start families -- a jittered regular 388 13-gon (‘_jittered_13gon‘) and a jittered two-ring layout (‘_jittered_two_ring‘) -- 389 each with sigma0 ̃ 0.3 and a distinct deterministic RNG seed, for ‘_N_GEN‘ 390 generations of a ‘_POPSIZE‘-sized population each. 391 2. Track every restart’s own best raw-scored (min triangle area / hull area) 392 configuration, not just the single overall-best across all restarts, so several 393 structurally different basins survive into the next phase. 394 3. Spend one further CMA-ES pass, re-seeded at the overall-best point with a much 395 smaller sigma0, to locally polish it. 396 4. Polish the overall best, the CMA-ES refine result, and several of the next-best 397 distinct restart endpoints with ‘_run_slp‘ -- a trust-region Sequential Linear 398 Programming / convex-concave procedure that exploits the fact that every one of 399 the 286 triangle areas is an EXACT bilinear function of the 26 coordinates, so 400 each outer iteration solves one small LP to first-order optimality instead of 401 relying on further stochastic sampling. 402 5. Run a deterministic iterated local search (small perturbation kick + ‘_run_slp‘ 403 re-polish, fixed shrinking noise schedule) around the best SLP-polished 404 incumbent, keeping it only if the true ratio improves. 405 6. Re-score every candidate (all restarts’ bests, the CMA-ES polish, every SLP 406 polish, the ILS result, and the safe regular-13-gon fallback) with the exact, 407 non-smoothed ‘_ratio‘, and return whichever scores highest -- so this pipeline 408 can only match or improve on its own safe fallback, and each new phase can only 409 match or improve on the phases before it, never regress. 410 Everything is fully deterministic (numpy ‘Generator‘ objects seeded with fixed 411 integers throughout, no unseeded randomness anywhere), so results are exactly 412 reproducible across runs and platforms. 413 75 414 Compute budget: sized (via empirical scratch-run timing of this exact CMA-ES/SLP 415 code) to use a large fraction of the evaluator’s headroom below its 360s timeout 416 while keeping a comfortable safety margin, since more CMA-ES restarts/generations 417 is the primary lever for escaping local optima on this rugged, non-smooth landscape. 418 419 Returns: 420 points: np.ndarray of shape (13,2) with the x,y coordinates of the points. 421 """ 422 _N_RESTARTS = 30 423 _POPSIZE = 64 424 _N_GEN = 800 425 _REFINE_GEN = 600 426 427 fallback = _regular_13gon() 428 candidates = [(_ratio(fallback), fallback)] 429 430 try: 431 restart_results = [] 432 for restart in range(_N_RESTARTS): 433 init_rng = np.random.default_rng(1000 + restart) 434 if restart % 2 == 0: 435 mean0 = _jittered_13gon(init_rng, jitter=0.05) 436 else: 437 mean0 = _jittered_two_ring(init_rng, m_outer=8, jitter=0.05) 438 x, score = _cma_es(mean0, sigma0=0.3, popsize=_POPSIZE, n_gen=_N_GEN, seed=restart) 439 restart_results.append((score, x)) 440 441 if restart_results: 442 restart_results.sort(key=lambda c: -c[0]) 443 overall_best_score, overall_best_x = restart_results[0] 444 445 pts = overall_best_x.reshape(_N, 2) 446 candidates.append((_ratio(pts), pts)) 447 448 # Local polish: re-run CMA-ES seeded at the best point found so far, 449 # with a much tighter initial step size, to squeeze out any further 450 # local improvement without risking a jump to a worse basin. 451 refine_x, refine_score = _cma_es( 452 overall_best_x, sigma0=0.05, popsize=_POPSIZE, n_gen=_REFINE_GEN, seed=999 453 ) 454 refine_pts = refine_x.reshape(_N, 2) 455 candidates.append((_ratio(refine_pts), refine_pts)) 456 457 # --- exact-bilinear trust-region SLP local polish ----------------------- 458 # Polish the overall best, the CMA-ES refine result, and several of the 459 # next-best distinct restart endpoints -- so if the single best restart 460 # happens to sit in a basin that is awkward for SLP’s linearization, a 461 # different restart’s basin still gets a chance to become the winner. 462 slp_seeds = [overall_best_x, refine_x] + [x for _, x in restart_results[1:8]] 463 best_slp_ratio, best_slp_x = -1.0, None 464 for seed_x in slp_seeds: 465 x_slp, r_slp = _run_slp(seed_x, max_outer=200, r0=0.2) 466 candidates.append((r_slp, x_slp.reshape(_N, 2))) 467 if r_slp > best_slp_ratio: 468 best_slp_ratio, best_slp_x = r_slp, x_slp 469 470 # --- deterministic iterated local search around the SLP incumbent ------ 471 # Fixed, shrinking-noise-scale kicks re-polished by ‘_run_slp‘; only kept 472 # if the true ratio improves, so this can only help, never regress. 473 if best_slp_x is not None: 474 kick_scales = (0.05, 0.03, 0.015, 0.008, 0.004, 0.002) 475 for it in range(120): 476 scale = kick_scales[it % len(kick_scales)] 477 ils_rng = np.random.default_rng(50000 + it) 478 kick = ils_rng.normal(0.0, scale, size=_DIM) 479 x0_ils = best_slp_x + kick 480 x_ils, r_ils = _run_slp(x0_ils, max_outer=200, r0=0.2) 481 if r_ils > best_slp_ratio: 482 best_slp_ratio, best_slp_x = r_ils, x_ils 483 candidates.append((best_slp_ratio, best_slp_x.reshape(_N, 2))) 484 except Exception: 485 pass 486 487 best_score, best_points = max(candidates, key=lambda c: c[0]) 488 if best_points is None or not np.isfinite(best_score) or best_score <= 0: 489 best_points = fallback 490 491 return best_points 76 K.2CLOUDCAST Cloudcast: base program 1 def search_algorithm(src, dsts, G, num_partitions): 2 EGRESS = ’aws’: 10.0, ’gcp’: 14.0, ’azure’: 32.0 3 4 h = G.copy() 5 h.remove_edges_from(list(h.in_edges(src)) + list(nx.selfloop_edges(h))) 6 h.remove_edges_from([(u, v) for u, v, d in h.edges(data=True) if d.get(’cost’) is None]) 7 8 bc_topology = BroadCastTopology(src, dsts, num_partitions) 9 if not dsts: 10 return bc_topology 11 12 sp_len, sp_pth = , 13 for u in list(h.nodes()): 14 try: 15 ls, ps = nx.single_source_dijkstra(h, u, weight=’cost’) 16 sp_len[u] = ls 17 sp_pth[u] = ps 18 except Exception: 19 sp_len[u] = 20 sp_pth[u] = 21 22 def gc(u, v): return sp_len.get(u, ).get(v, float(’inf’)) 23 def gp(u, v): return sp_pth.get(u, ).get(v) 24 25 def add_path(dst, seq): 26 for i in range(len(seq) - 1): 27 s, t = seq[i], seq[i + 1] 28 if G.has_edge(s, t): 29 # Use G’s real cost/throughput -- this used to be overwritten to cost=0.0, 30 # throughput=1e12 here, which faked a near-zero total_cost regardless of the 31 # path actually chosen. See docs/GPT56_TERRA_EVALUATOR_HARDENING.md. 32 ed = dict(G[s][t]) 33 for j in range(num_partitions): 34 bc_topology.append_dst_partition_path(dst, j, [s, t, ed]) 35 36 def is_assigned(dst): 37 return all(bc_topology.paths.get(dst, ).get(str(j)) is not None 38 for j in range(num_partitions)) 39 40 src_egress = EGRESS.get(src.split(’:’)[0], 10.0) 41 N = len(dsts) 42 dst_list = list(dsts) 43 INF = float(’inf’) 44 45 # Azure nodes sorted by proximity to src 46 azure_nodes = sorted( 47 [n for n in h.nodes() if n.split(’:’)[0] == ’azure’ and gc(src, n) < INF], 48 key=lambda n: gc(src, n) 49 ) 50 51 best_score = INF # actual_nodes/bottleneck -- lower is better 52 best_exec = None 53 54 # -- Single-hub plans ----------------------------------------------------- 55 for hub in azure_nodes: 56 hub_is_dst = hub in dsts 57 others = [d for d in dst_list if d != hub] 58 Knh = len(others) 59 60 hp = gp(src, hub) 61 if not hp: 62 continue 63 64 for r in range(1, Knh + 1): 65 src_out = 1 + (Knh - r) 66 sf = src_egress / src_out 67 hf = 32.0 / r 68 bn = min(sf, hf) 69 70 reachable = sorted([d for d in others if gc(hub, d) < INF], 71 key=lambda d: gc(hub, d)) 72 if len(reachable) < r: 73 continue 74 75 hub_dsts = reachable[:r] 76 direct_dsts = [d for d in others if d not in hub_dsts] 77 if not all(gc(src, d) < INF for d in direct_dsts): 78 continue 79 80 # Count actual path nodes for accurate cost estimation 81 nodes = set(hp) 82 for d in hub_dsts: 83 dp = gp(hub, d) 84 if dp: 85 nodes.update(dp) 77 86 for d in direct_dsts: 87 dp = gp(src, d) 88 if dp: 89 nodes.update(dp) 90 91 score = len(nodes) / bn if bn > 0 else INF 92 if score < best_score: 93 best_score = score 94 _hub, _hid = hub, hub_is_dst 95 _hd, _d = list(hub_dsts), list(direct_dsts) 96 97 def exec_1hub(hub=_hub, hid=_hid, hd=_hd, d=_d): 98 hp = gp(src, hub) 99 if hp: 100 if hid: 101 add_path(hub, hp) 102 for d in hd: 103 dp = gp(hub, d) 104 if dp: 105 add_path(d, hp + dp[1:]) 106 for d in d: 107 dp = gp(src, d) 108 if dp: 109 add_path(d, dp) 110 111 best_exec = exec_1hub 112 113 # -- Two-hub plans -------------------------------------------------------- 114 # Allows both dst and non-dst Azure nodes as hubs. 115 # Key gain: two Azure dsts as hubs -> src has only 2 out-edges (full egress/2 116 # per hub), each hub fans out with 32 Gbps egress, no extra relay nodes. 117 top_azure = azure_nodes[:15] 118 for i in range(len(top_azure)): 119 h1 = top_azure[i] 120 h1_is_dst = h1 in dsts 121 hp1 = gp(src, h1) 122 if not hp1: 123 continue 124 125 for j in range(i + 1, len(top_azure)): 126 h2 = top_azure[j] 127 h2_is_dst = h2 in dsts 128 hp2 = gp(src, h2) 129 if not hp2: 130 continue 131 132 # Dsts that need routing (excluding h1 and h2 themselves) 133 others = [d for d in dst_list if d != h1 and d != h2] 134 Mo = len(others) 135 136 # src always has exactly 2 out-edges: one to h1, one to h2 137 sf = src_egress / 2 138 139 for r1 in range(0 if h1_is_dst else 1, Mo + 1): 140 r2 = Mo - r1 141 # h2 must relay at least one other unless it’s a dst itself 142 if r2 == 0 and not h2_is_dst: 143 continue 144 145 hf1 = 32.0 / r1 if r1 > 0 else INF 146 hf2 = 32.0 / r2 if r2 > 0 else INF 147 bn = min(sf, hf1, hf2) 148 149 reach1 = sorted([d for d in others if gc(h1, d) < INF], 150 key=lambda d: gc(h1, d)) 151 if len(reach1) < r1: 152 continue 153 h1_others = reach1[:r1] 154 h2_others = [d for d in others if d not in h1_others] 155 if any(gc(h2, d) >= INF for d in h2_others): 156 continue 157 158 nodes = set(hp1) | set(hp2) 159 for d in h1_others: 160 dp = gp(h1, d) 161 if dp: 162 nodes.update(dp) 163 for d in h2_others: 164 dp = gp(h2, d) 165 if dp: 166 nodes.update(dp) 167 168 score = len(nodes) / bn if bn > 0 else INF 169 if score < best_score: 170 best_score = score 171 _h1, _h2 = h1, h2 172 _h1id, _h2id = h1_is_dst, h2_is_dst 173 _h1o, _h2o = list(h1_others), list(h2_others) 174 175 def exec_2hub(h1=_h1, h2=_h2, h1id=_h1id, h2id=_h2id, 176 h1o=_h1o, h2o=_h2o): 78 177 hp1 = gp(src, h1) 178 hp2 = gp(src, h2) 179 if hp1: 180 if h1id: 181 add_path(h1, hp1) 182 for d in h1o: 183 dp = gp(h1, d) 184 if dp: 185 add_path(d, hp1 + dp[1:]) 186 if hp2: 187 if h2id: 188 add_path(h2, hp2) 189 for d in h2o: 190 dp = gp(h2, d) 191 if dp: 192 add_path(d, hp2 + dp[1:]) 193 194 best_exec = exec_2hub 195 196 if best_exec: 197 best_exec() 198 199 # -- Arborescence fallback for any unassigned destinations ----------------- 200 unassigned = [d for d in dsts if not is_assigned(d)] 201 if unassigned: 202 try: 203 all_terminals = [src] + list(dsts) 204 metric_G = nx.DiGraph() 205 sp_cache = 206 for u in all_terminals: 207 for v in all_terminals: 208 if u == v: 209 continue 210 c = gc(u, v) 211 if c < INF: 212 metric_G.add_edge(u, v, cost=c) 213 sp_cache[(u, v)] = gp(u, v) 214 metric_G.remove_edges_from(list(metric_G.in_edges(src))) 215 arb = nx.minimum_spanning_arborescence(metric_G, attr=’cost’) 216 for d in unassigned: 217 if is_assigned(d): 218 continue 219 try: 220 vp = nx.shortest_path(arb, src, d) 221 actual = [] 222 for i in range(len(vp) - 1): 223 seg = sp_cache.get((vp[i], vp[i + 1])) 224 if not seg: 225 actual = None 226 break 227 actual = actual + seg[1:] if actual else list(seg) 228 if actual: 229 add_path(d, actual) 230 else: 231 p = gp(src, d) 232 if p: 233 add_path(d, p) 234 except Exception: 235 p = gp(src, d) 236 if p: 237 add_path(d, p) 238 except Exception: 239 for d in unassigned: 240 if not is_assigned(d): 241 p = gp(src, d) 242 if p: 243 add_path(d, p) 244 245 return bc_topology Cloudcast: final evolved program 1 def search_algorithm(src, dsts, G, num_partitions): 2 """ 3 Three-candidate broadcast router, ranked by a proxy that replays the 4 * actual * BCSimulator cost model (egress $ + throttle-aware instance-time 5 $) instead of a naive edge-cost sum. 6 7 Why a proxy simulator, not just summed edge cost: BCSimulator.__total_cost 8 = sum_egress_cost + sum_instance_cost. sum_egress_cost only depends on the 9 * set * of edges used (dedup’d), so summed real cost over the union of edges 10 is exact for that term. But sum_instance_cost = (#nodes touched) * num_vms 11 * ($/instance-hr) * runtime, and runtime = partition_data_vol / bottleneck 12 -- and BCSimulator.__construct_g * silently shrinks * an edge’s usable flow 13 below its nominal throughput whenever the SUM OF RAW THROUGHPUTS of all 14 edges touching a node exceeds that node’s provider ingress/egress limit 15 (proportional, equal-split throttling). A tree that is cheapest in pure 16 $/GB terms can still fan out through a node with many/large edges, get 79 17 throttled hard, and pay a much larger instance-time bill than a 18 slightly-more-expensive-in-$ tree that avoids that bottleneck. Both 19 real_dollar_cost’s blindness to this AND a pure min-cost-flow’s blindness 20 to $ cost are individually wrong; ranking every candidate by a full 21 proxy replay of the simulator (using a data_vol=1 convention -- both cost 22 terms scale linearly in data_vol for a fixed num_partitions, so the 23 * ranking * among same-config candidates is data_vol-invariant, letting us 24 drop the unknown real data_vol/num_vms entirely and use the known eval 25 harness constants: num_vms=2, cost_per_instance_hr=0.54, and the provider 26 ingress/egress defaults every example config pins to those same values) 27 fixes both blind spots at once. 28 29 Candidate A (terminal-only metric-closure arborescence): all-pairs 30 cheapest-path cost among src union dsts only, nx.minimum_spanning_arborescence 31 rooted at src, realized via real shortest paths. Restricting the 32 arborescence’s node set to terminals-only (rather than all graph nodes) 33 empirically gives the lowest cost for this construction, since MSA 34 computes a * spanning * tree that must reach every node handed to it. 35 36 Candidate B (best single unrestricted external relay): patches Candidate 37 A’s structural blind spot -- it can only reuse a shared prefix that lies on 38 the * specific * shortest path between two terminals, never a non-terminal 39 relay chosen purely for its cheap fan-out to many destinations at once. 40 O(|V|) sweep over every node as a src->hub->dsts relay. 41 42 Candidate C (capacitated min-cost flow, node-split): the one degree of 43 freedom neither A nor B has -- explicit, LP-optimal * capacity-respecting * 44 routing. Each node is split into an #in/#out pair joined by an edge whose 45 capacity is min(ingress_limit, egress_limit) for that node’s provider (the 46 same node-throttling resource A/B are blind to); each real graph edge 47 becomes an #out->#in edge with capacity=throughput, weight=cost; a super 48 source feeds src with one demand unit per destination and each 49 destination’s #in node drains to a super sink. nx.max_flow_min_cost solves 50 this once, and the resulting flow is decomposed into src->dst paths by 51 repeatedly extracting a positive-residual path per destination (pure 52 bookkeeping, not a second optimization) and collapsing each node’s #in/#out 53 split back to the real node. Any destination the flow leaves unmet (can 54 happen since it is one * joint * solve across all destinations, sharing 55 demand against shared capacity) is patched with a direct shortest path so 56 every candidate is always a complete, fairly-comparable assignment. 57 58 Candidate D (Dreyfus-Wagner bitmask DP, lexicographic composite weight): 59 exact rooted directed Steiner-tree DP over dp[mask][v] where every edge’s 60 weight is a single composite scalar cost * BIG_M + 1/throughput (BIG_M=1e6). 61 A/B/C above already provably hit the global-minimum egress cost on every 62 benchmark config, so the composite’s cost * BIG_M term guarantees this DP 63 reproduces that same optimum bit-for-bit (real cost deltas always dwarf 64 the secondary term by >>1e6x); among ties in that optimum, the added 65 1/throughput term steers the DP’s own merge/relay tie-breaking toward 66 fewer relay hops through higher-throughput links -- the same levers that 67 drive the (otherwise-unexploited) instance-cost term down. 68 69 Candidate E (effective-resistance tie-break) and Candidate F (Hamiltonian 70 TSP relay chain, bounding fan-out to 1 at every internal terminal by 71 construction) are defined further below, alongside A-D. 72 73 All candidates are scored with the same simulate_cost() proxy and the 74 cheapest wins; any candidate can only help since none is favored a 75 priori. 76 """ 77 h = G.copy() 78 h.remove_edges_from(list(h.in_edges(src)) + list(nx.selfloop_edges(h))) 79 h.remove_edges_from([(u, v) for u, v, d in h.edges(data=True) if d.get(’cost’) is None]) 80 81 bc_topology = BroadCastTopology(src, dsts, num_partitions) 82 if not dsts: 83 return bc_topology 84 85 INF = float(’inf’) 86 terminals = [src] + list(dsts) 87 88 # Shortest paths from every node in h (not just terminals): needed so the 89 # single-relay candidate below can evaluate gc(hub, d)/gp(hub, d) for an 90 # arbitrary non-terminal hub, not only for src and the destinations. 91 sp_len, sp_pth = , 92 for u in list(h.nodes()): 93 try: 94 ls, ps = nx.single_source_dijkstra(h, u, weight=’cost’) 95 sp_len[u] = ls 96 sp_pth[u] = ps 97 except Exception: 98 sp_len[u] = 99 sp_pth[u] = 100 101 def gc(u, v): return sp_len.get(u, ).get(v, INF) 102 def gp(u, v): return sp_pth.get(u, ).get(v) 103 104 def add_path(dst, seq): 105 for i in range(len(seq) - 1): 106 s, t = seq[i], seq[i + 1] 107 if G.has_edge(s, t): 80 108 # Use G’s real cost/throughput -- this used to be overwritten to cost=0.0, 109 # throughput=1e12 here, which faked a near-zero total_cost regardless of the 110 # path actually chosen. See docs/GPT56_TERRA_EVALUATOR_HARDENING.md. 111 ed = dict(G[s][t]) 112 for j in range(num_partitions): 113 bc_topology.append_dst_partition_path(dst, j, [s, t, ed]) 114 115 def is_assigned(dst): 116 return all(bc_topology.paths.get(dst, ).get(str(j)) is not None 117 for j in range(num_partitions)) 118 119 def path_edges(nodes): 120 return [(nodes[i], nodes[i + 1]) for i in range(len(nodes) - 1)] 121 122 # Eval-harness constants (see simulator.py / evaluator_hardened.py): num_vms 123 # is hardcoded to 2 by the harness, and every example config’s ingress/ 124 # egress overrides are exactly the simulator’s own defaults, so the 125 # * effective * (post-num_vms-scaling) limits are fixed and known here. 126 NUM_VMS = 2 127 COST_PER_INSTANCE_HR = 0.54 128 INGRESS_LIMITS = "aws": 10 * NUM_VMS, "gcp": 16 * NUM_VMS, "azure": 16 * NUM_VMS 129 EGRESS_LIMITS = "aws": 5 * NUM_VMS, "gcp": 7 * NUM_VMS, "azure": 16 * NUM_VMS 130 131 def simulate_cost(assignment): 132 """Proxy replay of BCSimulator.__construct_g/__transfer_time/__total_cost 133 for a dst: [node..node] assignment (same path for every partition of 134 a destination, matching add_path’s convention), using a data_vol=1 135 convention. Both real cost terms scale linearly in data_vol for a fixed 136 num_partitions, so ranking candidates by this proxy is equivalent to 137 ranking by the simulator’s real total_cost for this config.""" 138 if not assignment or len(assignment) != len(dsts): 139 return INF 140 g = nx.DiGraph() 141 for p in assignment.values(): 142 if not p: 143 return INF 144 for u, v in path_edges(p): 145 if not h.has_edge(u, v): 146 return INF 147 if not g.has_edge(u, v): 148 tp = h[u][v][’throughput’] 149 g.add_edge(u, v, throughput=tp, cost=h[u][v][’cost’], flow=tp, partitions=set()) 150 g[u][v][’partitions’].update(range(num_partitions)) 151 152 # Proportional, equal-split throttling -- exactly mirrors __construct_g. 153 for node in g.nodes(): 154 provider = node.split(’:’)[0] 155 in_edges, out_edges = list(g.in_edges(node)), list(g.out_edges(node)) 156 in_sum = sum(g[u][v][’flow’] for u, v in in_edges) 157 out_sum = sum(g[u][v][’flow’] for u, v in out_edges) 158 ing_lim = INGRESS_LIMITS.get(provider, INF) 159 egr_lim = EGRESS_LIMITS.get(provider, INF) 160 if in_edges and in_sum > ing_lim: 161 share = ing_lim / len(in_edges) 162 for u, v in in_edges: 163 g[u][v][’flow’] = min(g[u][v][’flow’], share) 164 if out_edges and out_sum > egr_lim: 165 share = egr_lim / len(out_edges) 166 for u, v in out_edges: 167 g[u][v][’flow’] = min(g[u][v][’flow’], share) 168 169 egress_cost = sum(len(d[’partitions’]) * (1.0 / num_partitions) * d[’cost’] 170 for _, _, d in g.edges(data=True)) 171 172 max_t = 0.0 173 for p in assignment.values(): 174 edges = path_edges(p) 175 bottleneck = min(g[u][v][’flow’] for u, v in edges) 176 if bottleneck <= 0: 177 return INF 178 max_t = max(max_t, (1.0 / num_partitions) / bottleneck) 179 180 instance_cost = g.number_of_nodes() * NUM_VMS * (COST_PER_INSTANCE_HR / 3600.0) * max_t 181 return egress_cost + instance_cost 182 183 def build_metric_closure_arborescence(node_set): 184 metric_G = nx.DiGraph() 185 sp_cache = 186 for u in node_set: 187 for v in node_set: 188 if u == v: 189 continue 190 c = gc(u, v) 191 if c < INF: 192 metric_G.add_edge(u, v, cost=c) 193 sp_cache[(u, v)] = gp(u, v) 194 metric_G.remove_edges_from(list(metric_G.in_edges(src))) 195 try: 196 arb = nx.minimum_spanning_arborescence(metric_G, attr=’cost’) 197 except Exception: 198 return None 81 199 assignment = 200 for d in dsts: 201 try: 202 vp = nx.shortest_path(arb, src, d) 203 actual = [] 204 for i in range(len(vp) - 1): 205 seg = sp_cache.get((vp[i], vp[i + 1])) 206 if not seg: 207 actual = None 208 break 209 actual = actual + seg[1:] if actual else list(seg) 210 if actual: 211 assignment[d] = actual 212 except Exception: 213 continue 214 return assignment 215 216 # -- Candidate A (primary): terminal-only metric-closure arborescence ------ 217 arb_assignment = build_metric_closure_arborescence(terminals) or 218 219 # -- Candidate B: best single unrestricted external relay ----------------- 220 # For every reachable node (any provider) other than src, try routing every 221 # destination through it (src -> hub -> dst), plus the hub’s own direct leg 222 # if the hub happens to be a destination itself. Rank by the full 223 # simulate_cost proxy so it is directly comparable to the other candidates. 224 best_hub_assignment, best_hub_cost = None, INF 225 for hub in h.nodes(): 226 if hub == src or gc(src, hub) >= INF: 227 continue 228 hp = gp(src, hub) 229 if not hp: 230 continue 231 others = [d for d in dsts if d != hub] 232 if any(gc(hub, d) >= INF for d in others): 233 continue 234 cand = 235 if hub in dsts: 236 cand[hub] = list(hp) 237 for d in others: 238 dp = gp(hub, d) 239 if dp: 240 cand[d] = hp + dp[1:] 241 if len(cand) != len(dsts): 242 continue 243 cost = simulate_cost(cand) 244 if cost < best_hub_cost: 245 best_hub_cost = cost 246 best_hub_assignment = cand 247 248 # -- Candidate C: capacitated min-cost flow with node-splitting ----------- 249 # Explicit, LP-optimal capacity-respecting routing -- the one degree of 250 # freedom A/B structurally lack. Each node n is split into n#in/n#out 251 # joined by a capacity=min(ingress,egress) edge (the same throttling 252 # resource simulate_cost’s throttling step models); real edges become 253 # #out->#in edges capacitated by throughput, weighted by cost; a super 254 # source feeds src with one demand unit per destination, and each 255 # destination’s #in drains to a super sink. One nx.max_flow_min_cost solve 256 # gives the joint routing; flow is then decomposed into per-destination 257 # paths by repeatedly walking a positive-residual S->dst path (bookkeeping, 258 # not re-optimizing) and collapsing #in/#out pairs back to real nodes. 259 flow_assignment = None 260 try: 261 SCALE = 10 # uniform int scale for capacities AND demand (units must match) 262 COST_SCALE = 1_000_000 263 F = nx.DiGraph() 264 for n in h.nodes(): 265 provider = n.split(’:’)[0] 266 cap = min(INGRESS_LIMITS.get(provider, INF), EGRESS_LIMITS.get(provider, INF)) 267 if cap == INF: 268 continue 269 F.add_edge(f"n#in", f"n#out", capacity=max(1, int(round(cap * SCALE))), weight=0) 270 for u, v, d in h.edges(data=True): 271 c, t = d.get(’cost’), d.get(’throughput’) 272 if c is None or t is None or t <= 0: 273 continue 274 if f"u#out" in F and f"v#in" in F: 275 F.add_edge(f"u#out", f"v#in", 276 capacity=max(1, int(round(t * SCALE))), weight=int(round(c * COST_SCALE))) 277 278 avail_dsts = [d for d in dsts if f"d#in" in F] 279 demand = SCALE # one demand unit per destination, same scale as capacities 280 if avail_dsts and f"src#in" in F: 281 F.add_edge(’S’, f"src#in", capacity=demand * len(avail_dsts), weight=0) 282 for d in avail_dsts: 283 F.add_edge(f"d#in", ’T’, capacity=demand, weight=0) 284 285 flow_dict = nx.max_flow_min_cost(F, ’S’, ’T’) 286 residual = (u, v): f for u, nbrs in flow_dict.items() for v, f in nbrs.items() if f > 0 287 288 def extract_path(target): 289 path, visited, cur = [’S’], ’S’, ’S’ 82 290 while cur != target: 291 nxt = next((v2 for (u2, v2), f in residual.items() 292 if u2 == cur and f > 0 and v2 not in visited), None) 293 if nxt is None: 294 return None 295 path.append(nxt) 296 visited.add(nxt) 297 cur = nxt 298 return path 299 300 flow_assignment = 301 for d in avail_dsts: 302 p = extract_path(f"d#in") 303 if p is None: 304 continue 305 # Cap consumption at this destination’s own demand share -- taking 306 # the full min-residual along the path would over-consume capacity 307 # that a shared upstream edge is also carrying for OTHER 308 # destinations, starving them even though the joint solve 309 # actually reserved enough flow for everyone. 310 amt = max(1, min([demand] + [residual[(p[i], p[i + 1])] for i in range(len(p) - 1)])) 311 for i in range(len(p) - 1): 312 residual[(p[i], p[i + 1])] -= amt 313 nodes_seq = [] 314 for node in p: 315 if node in (’S’, ’T’): 316 continue 317 base = node.rsplit(’#’, 1)[0] 318 if not nodes_seq or nodes_seq[-1] != base: 319 nodes_seq.append(base) 320 if (nodes_seq and nodes_seq[0] == src and nodes_seq[-1] == d 321 and all(h.has_edge(nodes_seq[i], nodes_seq[i + 1]) for i in range(len(nodes_seq) - 1))): 322 flow_assignment[d] = nodes_seq 323 324 # Any destination the joint flow solve left unmet (shared demand vs 325 # shared capacity can starve a costly sink) is patched with a direct 326 # shortest path so Candidate C is always complete and fairly 327 # comparable to A/B. 328 for d in dsts: 329 if d not in (flow_assignment or ): 330 dp = gp(src, d) 331 if dp: 332 flow_assignment = flow_assignment or 333 flow_assignment[d] = dp 334 if flow_assignment is not None and len(flow_assignment) != len(dsts): 335 flow_assignment = None 336 except Exception: 337 flow_assignment = None 338 339 # -- Candidate D: Dreyfus-Wagner bitmask Steiner-tree DP, lexicographic ---- 340 # bi-objective composite edge weighting. 341 # 342 # Independent verification (from-scratch DP + all-pairs Dijkstra by $/GB 343 # cost, cross-checked against candidates A/B/C above) confirms every prior 344 # candidate already lands on the exact global minimum of the egress-cost 345 # term (sum of distinct-edge $/GB costs over the union broadcast tree) on 346 # every benchmark config -- there is zero remaining headroom on the 347 # primary objective, which is 99.97%+ of total_cost. The only unexploited 348 # lever is that whenever multiple Steiner trees tie on that minimum egress 349 # cost, every prior candidate’s construction (MST-of-metric-closure, 350 # single-relay sweep, min-cost-flow decomposition) picks an ARBITRARY 351 # member of that tie set -- none of them explicitly steers the choice 352 # toward the (negligible but nonzero) instance-cost term. 353 # 354 # Rather than a two-stage pipeline (solve egress-optimal, then re-optimize 355 # instance-cost among ties), this folds both objectives into ONE DP pass: 356 # every edge’s weight is a single composite scalar 357 # w(u, v) = cost(u, v) * BIG_M + 1.0 / throughput(u, v) 358 # with BIG_M = 1e6 chosen so that any two Steiner trees differing at all 359 # in real $/GB egress cost (granularity of cents/fractions-of-a-cent) are 360 # separated by >>1e6x more composite weight than any plausible secondary 361 # (1/throughput) sum could ever close -- so the DP can never trade a 362 # fractional-cent egress increase for a secondary-objective gain, and the 363 # primary optimum is preserved bit-for-bit. Among true primary ties, the 364 # 1/throughput term steers the same merge/relay recurrence toward fewer 365 # relay hops through higher-throughput links, which is exactly what drives 366 # down both the node-count and bottleneck-runtime factors of instance_cost. 367 # 368 # Implementation: classic small-terminal/large-node Dreyfus-Wagner bitmask 369 # DP over dp[mask][v] (mask = subset of destinations, v = any node in h), 370 # built on all-pairs shortest * composite * distances from a single 371 # scipy.sparse.csgraph.dijkstra(directed=True) call (vectorized over all 372 # sources at once, avoiding k separate networkx Dijkstra calls). Capped at 373 # a modest terminal count (state space is O(2ˆk), split step O(3ˆk * n)) 374 # since k is small (single digits) on this benchmark’s configs; returns 375 # None (never a regression, since it’s just one more entry in ‘candidates‘ 376 # below) if scipy is unavailable, too many terminals, or any destination 377 # is unreachable. 378 def build_dp_composite_assignment(max_terminals=10): 379 try: 83 380 import numpy as np 381 from scipy.sparse import csr_matrix 382 from scipy.sparse.csgraph import dijkstra 383 except ImportError: 384 return None 385 386 terms = list(dsts) 387 k = len(terms) 388 if k == 0 or k > max_terminals: 389 return None 390 391 nodes = list(h.nodes()) 392 if src not in nodes or any(t not in nodes for t in terms): 393 return None 394 idx = node: i for i, node in enumerate(nodes) 395 n = len(nodes) 396 src_i = idx[src] 397 term_idx = [idx[t] for t in terms] 398 399 BIG_M = 1_000_000.0 400 rows, cols, data = [], [], [] 401 for u, v, d in h.edges(data=True): 402 c, tp = d.get(’cost’), d.get(’throughput’) 403 if c is None or not tp or tp <= 0: 404 continue 405 rows.append(idx[u]); cols.append(idx[v]); data.append(c * BIG_M + 1.0 / tp) 406 if not rows: 407 return None 408 csgraph = csr_matrix((data, (rows, cols)), shape=(n, n)) 409 410 try: 411 dist, pred = dijkstra(csgraph, directed=True, indices=None, return_predecessors=True) 412 except Exception: 413 return None 414 if any(dist[src_i, t] == INF for t in term_idx): 415 return None 416 417 full_mask = (1 << k) - 1 418 dp = [np.full(n, INF) for _ in range(1 << k)] 419 choice = [[None] * n for _ in range(1 << k)] 420 for bi in range(k): 421 dp[1 << bi] = dist[:, term_idx[bi]].copy() 422 423 masks_by_size = sorted(range(3, full_mask + 1), key=lambda m: bin(m).count(’1’)) 424 for mask in masks_by_size: 425 if mask & (mask - 1) == 0: 426 continue # singleton, already a base case 427 best, best_choice = dp[mask], choice[mask] 428 sub = (mask - 1) & mask 429 while sub > 0: 430 other = mask ˆ sub 431 if sub < other: # each unordered split considered once 432 cand = dp[sub] + dp[other] 433 improved = cand < best 434 if np.any(improved): 435 for vi in np.nonzero(improved)[0]: 436 best_choice[vi] = (’split’, sub, other) 437 best = np.where(improved, cand, best) 438 sub = (sub - 1) & mask 439 # Vectorized "extend": relay from every v through its single best u. 440 relay = dist + best[np.newaxis, :] 441 relay_u = np.argmin(relay, axis=1) 442 relay_val = relay[np.arange(n), relay_u] 443 improved = relay_val < best 444 if np.any(improved): 445 for vi in np.nonzero(improved)[0]: 446 best_choice[vi] = (’relay’, int(relay_u[vi])) 447 best = np.where(improved, relay_val, best) 448 dp[mask] = best 449 450 if dp[full_mask][src_i] == INF: 451 return None 452 453 def path_node_idxs(u_i, v_i): 454 if u_i == v_i: 455 return [u_i] 456 path, cur = [v_i], v_i 457 while cur != u_i: 458 p = int(pred[u_i, cur]) 459 if p < 0: 460 return None 461 path.append(p) 462 cur = p 463 path.reverse() 464 return path 465 466 edge_idx_set = set() 467 468 def collect(v_i, mask): 469 if mask & (mask - 1) == 0: 470 t = term_idx[mask.bit_length() - 1] 84 471 pn = path_node_idxs(v_i, t) 472 if pn is None: 473 raise ValueError("unreachable base case") 474 for i in range(len(pn) - 1): 475 edge_idx_set.add((pn[i], pn[i + 1])) 476 return 477 ch = choice[mask][v_i] 478 if ch is None: 479 raise ValueError("no recorded choice") 480 if ch[0] == ’split’: 481 collect(v_i, ch[1]) 482 collect(v_i, ch[2]) 483 else: 484 _, u_i = ch 485 if u_i != v_i: 486 pn = path_node_idxs(v_i, u_i) 487 if pn is None: 488 raise ValueError("unreachable relay") 489 for i in range(len(pn) - 1): 490 edge_idx_set.add((pn[i], pn[i + 1])) 491 collect(u_i, mask) 492 493 try: 494 collect(src_i, full_mask) 495 except ValueError: 496 return None 497 498 tree = nx.DiGraph() 499 for (ui, vi) in edge_idx_set: 500 u, v = nodes[ui], nodes[vi] 501 if h.has_edge(u, v): 502 tree.add_edge(u, v) 503 504 assignment = 505 for d in terms: 506 try: 507 assignment[d] = nx.shortest_path(tree, src, d) 508 except Exception: 509 continue 510 return assignment 511 512 dp_assignment = build_dp_composite_assignment() 513 514 # -- Candidate E: effective-resistance / current-flow tie-break over the -- 515 # tight-edge (src-rooted shortest-path-DAG) subgraph. 516 # 517 # A-D above already provably land on the global-minimum egress-cost term 518 # (verified independently by MST, exact bitmask DP, and hillclimb 519 # variants alike -- see module docstring). Whenever more than one edge 520 # lies on an equal-cost shortest path from src (a genuine tie in the 521 # $/GB metric, not merely a modeling artifact), every prior candidate’s 522 # construction (MST-of-metric-closure, single-relay sweep, min-cost-flow 523 # decomposition, composite-weight DP) picks * an * arbitrary tree from that 524 # tie set -- none of them looks at how well the tied edges support 525 # parallel, low-bottleneck flow. This candidate instead treats the 526 # tight-edge subgraph as a resistor network: conductance(u, v) = 527 # throughput(u, v) (a wider pipe conducts more "current" for the same 528 # potential drop, i.e. lower effective resistance = less of a runtime 529 # bottleneck), builds the weighted graph Laplacian over that subgraph, 530 # and takes its Moore-Penrose pseudo-inverse (np.linalg.pinv) to get the 531 # exact pairwise effective resistance between src and every destination 532 # -- a continuous, physically-grounded proxy for the (otherwise 533 # unexploited) instance-cost/runtime term, since parallel tied paths 534 # combine like parallel resistors and lower effective resistance 535 # directly implies a higher-bottleneck-throughput route. For each 536 # destination, a unit current is injected at src and drained at that 537 # destination (phi = pinv(L) @ (e_src - e_dst)); walking from src and 538 # always following the tight edge carrying the most current 539 # (conductance * potential-drop) selects the specific spanning path 540 # within the tie set that best supports low-resistance parallel flow, 541 # rather than an MST’s structurally-arbitrary tie-break. 542 def build_current_flow_assignment(tol=1e-9): 543 try: 544 import numpy as np 545 except ImportError: 546 return None 547 s_dist = sp_len.get(src, ) 548 if not s_dist: 549 return None 550 # Tight edges: those satisfying the shortest-path triangle inequality 551 # with equality (within tol) -- exactly the edges that can appear in 552 # * some * src-rooted min-$/GB tree, i.e. the tie set to disambiguate. 553 tight_nodes = src 554 tight_edges = [] 555 for u, v, ed in h.edges(data=True): 556 du, dv = s_dist.get(u), s_dist.get(v) 557 if du is None or dv is None or du >= INF or dv >= INF: 558 continue 559 c, tp = ed.get(’cost’), ed.get(’throughput’) 560 if c is None or not tp or tp <= 0: 561 continue 85 562 if abs((du + c) - dv) <= tol * max(1.0, abs(dv)): 563 tight_edges.append((u, v, float(tp))) 564 tight_nodes.add(u) 565 tight_nodes.add(v) 566 if not tight_edges or any(d not in tight_nodes for d in dsts): 567 return None 568 569 nodes = sorted(tight_nodes) 570 idx = node: i for i, node in enumerate(nodes) 571 n = len(nodes) 572 cond = 573 out_adj = 574 for u, v, tp in tight_edges: 575 i, j = idx[u], idx[v] 576 cond[(i, j)] = cond.get((i, j), 0.0) + tp 577 out_adj.setdefault(u, []).append((v, tp)) 578 L = np.zeros((n, n)) 579 for (i, j), c in cond.items(): 580 L[i, i] += c 581 L[j, j] += c 582 L[i, j] -= c 583 L[j, i] -= c 584 try: 585 Lp = np.linalg.pinv(L) 586 except Exception: 587 return None 588 589 s_i = idx[src] 590 assignment = 591 for dnode in dsts: 592 d_i = idx[dnode] 593 rhs = np.zeros(n) 594 rhs[s_i] = 1.0 595 rhs[d_i] = -1.0 596 phi = Lp @ rhs # nodal potentials for unit src->dnode current 597 path, cur, visited = [src], src, src 598 ok = True 599 while cur != dnode: 600 nbrs = [(v, tp) for v, tp in out_adj.get(cur, []) if v not in visited] 601 if not nbrs: 602 ok = False 603 break 604 cur_i = idx[cur] 605 best_v, best_flow = None, 0.0 606 for v, tp in nbrs: 607 flow = tp * (phi[cur_i] - phi[idx[v]]) 608 if flow > best_flow + 1e-12: 609 best_flow, best_v = flow, v 610 if best_v is None: 611 ok = False 612 break 613 path.append(best_v) 614 visited.add(best_v) 615 cur = best_v 616 if not ok: 617 return None 618 assignment[dnode] = path 619 return assignment 620 621 cf_assignment = build_current_flow_assignment() 622 623 # -- Candidate F: Hamiltonian relay chain via TSP approximation ----------- 624 # 625 # A-E above are all tree/flow/resistor-network topologies that fan out at 626 # internal nodes: an ancestor forwards to potentially many children in 627 # one step. simulate_cost’s own throttling model (mirrored from 628 # __construct_g) proportionally * shrinks * each edge’s usable flow when 629 # the raw throughputs summed over all edges touching a node exceed that 630 # node’s provider ingress/egress limit -- so any node with out-degree > 1 631 # in the chosen tree pays this equal-split throttling tax on every one of 632 # its outgoing legs. This candidate is structurally immune to that tax by 633 # construction: it visits src union dsts in a single Hamiltonian order (a 634 # relay chain) where every terminal except the very last has out-degree 635 # exactly 1 -- there is no fan-out to throttle, at the cost of a 636 # potentially longer aggregate path length (more relay hops / more total 637 # $/GB before reaching the far end of the chain). 638 # 639 # The order itself is chosen by networkx’s TSP approximation 640 # (traveling_salesman_problem, greedy_tsp method) over the * metric 641 # closure * of the terminal set: a complete auxiliary graph where 642 # edge_weight(u, v) = min(gc(u, v), gc(v, u)), reusing the all-pairs 643 # Dijkstra table (gc/gp) already computed above rather than a second 644 # shortest-path solve. greedy_tsp needs a complete undirected graph, so 645 # the tour weight is symmetrized (min of both directions); this only 646 # affects which ordering the heuristic prefers, not the actual routing -- 647 # every realized leg below still uses the correct * directed * shortest 648 # path via gp(a, b) for that specific consecutive pair. cycle=True is 649 # requested (so greedy_tsp gets to optimize a genuine Hamiltonian cycle 650 # instead of the weaker Hamiltonian-path formulation) and the resulting 651 # closed tour is then rotated so it starts at src and the closing edge 652 # back to src is dropped, giving a strict permutation 86 653 # src -> d_i1 -> d_i2 -> ... -> d_in rooted at src as required. 654 def build_tsp_relay_assignment(): 655 try: 656 from networkx.algorithms.approximation import ( 657 traveling_salesman_problem, greedy_tsp, 658 ) 659 except ImportError: 660 return None 661 if len(terminals) < 2: 662 return None 663 664 metric_G = nx.Graph() 665 BIG = 1e15 666 for i, u in enumerate(terminals): 667 for v in terminals[i + 1:]: 668 w = min(gc(u, v), gc(v, u)) 669 metric_G.add_edge(u, v, cost=BIG if w >= INF else w) 670 if metric_G.number_of_nodes() != len(terminals): 671 return None 672 673 try: 674 cycle = traveling_salesman_problem( 675 metric_G, weight=’cost’, nodes=terminals, cycle=True, method=greedy_tsp) 676 except Exception: 677 return None 678 if not cycle or len(cycle) < 2: 679 return None 680 681 body = cycle[:-1] if cycle[0] == cycle[-1] else cycle 682 if src not in body or len(body) != len(terminals): 683 return None 684 si = body.index(src) 685 order = body[si:] + body[:si] # rotate so src leads the chain 686 687 assignment = 688 running = [src] 689 for i in range(len(order) - 1): 690 a, b = order[i], order[i + 1] 691 seg = gp(a, b) 692 if not seg: 693 return None 694 running = running + list(seg[1:]) 695 assignment[b] = list(running) 696 if len(assignment) != len(dsts) or set(assignment) != set(dsts): 697 return None 698 return assignment 699 700 tsp_relay_assignment = build_tsp_relay_assignment() 701 702 # -- Pick whichever full-coverage candidate scores cheapest --------------- 703 candidates = [] 704 if arb_assignment and len(arb_assignment) == len(dsts): 705 candidates.append((simulate_cost(arb_assignment), arb_assignment)) 706 if best_hub_assignment is not None: 707 candidates.append((best_hub_cost, best_hub_assignment)) 708 if flow_assignment is not None: 709 candidates.append((simulate_cost(flow_assignment), flow_assignment)) 710 if dp_assignment and len(dp_assignment) == len(dsts): 711 candidates.append((simulate_cost(dp_assignment), dp_assignment)) 712 if cf_assignment and len(cf_assignment) == len(dsts): 713 candidates.append((simulate_cost(cf_assignment), cf_assignment)) 714 if tsp_relay_assignment and len(tsp_relay_assignment) == len(dsts): 715 candidates.append((simulate_cost(tsp_relay_assignment), tsp_relay_assignment)) 716 717 if candidates: 718 _, chosen = min(candidates, key=lambda t: t[0]) 719 for d, p in chosen.items(): 720 add_path(d, p) 721 722 # -- Direct shortest-path fallback for anything still unassigned ----------- 723 unassigned = [d for d in dsts if not is_assigned(d)] 724 for d in unassigned: 725 p = gp(src, d) 726 if p: 727 add_path(d, p) 728 729 return bc_topology K.3TXN TXN: base program 1 2 def get_best_schedule(workload, num_seqs): 3 """ 4 Tabu search: swap+or-opt-1 full neighborhood (sampled for large n), 5 transaction-based tabu list, aspiration criterion, double-bridge restarts. 87 6 Initialization: Copeland + MDS + AP + multi-start greedy (proven best init). 7 """ 8 import math, time 9 import numpy as np 10 11 n = workload.num_txns 12 t0 = time.time() 13 tlimit = 170.0 14 15 def timed_out(frac=1.0): 16 return (time.time() - t0) >= tlimit * frac 17 18 def cost(seq): 19 return workload.get_opt_seq_cost(seq) 20 21 # Build pairwise cost matrix + Copeland wins 22 D = [[0.0] * n for _ in range(n)] 23 wins = [0] * n 24 for i in range(n): 25 for j in range(i + 1, n): 26 cij = cost([i, j]); cji = cost([j, i]) 27 D[i][j] = cij; D[j][i] = cji 28 if cij <= cji: wins[i] += 1 29 else: wins[j] += 1 30 copeland = sorted(range(n), key=lambda x: -wins[x]) 31 32 # 1D MDS: symmetrize, double-center, sort by dominant eigenvector 33 S = np.array([[(D[i][j] + D[j][i]) / 2.0 for j in range(n)] for i in range(n)]) 34 H = np.eye(n) - np.ones((n, n)) / n 35 B = -0.5 * (H @ S @ H) 36 _, evecs = np.linalg.eigh(B) 37 coord = evecs[:, -1] 38 mds_asc = list(np.argsort(coord)) 39 mds_desc = list(np.argsort(-coord)) 40 41 # AP relaxation 42 ap_seq = None 43 try: 44 from scipy.optimize import linear_sum_assignment 45 Dm = np.array(D, dtype=float); np.fill_diagonal(Dm, 1e12) 46 _, col = linear_sum_assignment(Dm) 47 visited = [False] * n; chains = [] 48 for s0 in range(n): 49 if visited[s0]: continue 50 ch = []; node = s0 51 while not visited[node]: 52 visited[node] = True; ch.append(node); node = int(col[node]) 53 chains.append(ch) 54 if len(chains) == 1: 55 ap_seq = chains[0][:] 56 else: 57 chains.sort(key=len, reverse=True); merged = list(chains[0]) 58 for ch in chains[1:]: 59 bc_ap = float(’inf’); bm = merged + ch 60 for fwd in [ch, ch[::-1]]: 61 for pos in range(len(merged) + 1): 62 trial = merged[:pos] + fwd + merged[pos:] 63 cv = cost(trial) 64 if cv < bc_ap: bc_ap = cv; bm = trial[:] 65 merged = bm 66 ap_seq = merged[:] 67 except Exception: 68 pass 69 70 # Greedy from top Copeland starters 71 def greedy_from(start): 72 s = [start]; rem = [x for x in range(n) if x != start] 73 while rem: 74 bc2, bt = float(’inf’), rem[0] 75 for t in rem: 76 nc = cost(s + [t]) 77 if nc < bc2: bc2, bt = nc, t 78 s.append(bt); rem.remove(bt) 79 return cost(s), s[:] 80 81 # Build best initial solution 82 best_c = float(’inf’); best_s = list(copeland) 83 for seq in ([mds_asc, mds_desc, copeland] + ([ap_seq] if ap_seq else [])): 84 c = cost(seq) 85 if c < best_c: best_c = c; best_s = seq[:] 86 for starter in copeland[:min(3, n)]: 87 if timed_out(0.22): break 88 gc, gs = greedy_from(starter) 89 if gc < best_c: best_c = gc; best_s = gs[:] 90 91 # Double-bridge perturbation for diversification restarts 92 def double_bridge(seq): 93 if n < 8: 94 p = seq[:] 95 l = random.randint(2, max(2, n // 2)) 96 a = random.randint(0, n - l) 88 97 p[a:a + l] = p[a:a + l][::-1] 98 return p 99 cuts = sorted(random.sample(range(1, n), 4)) 100 a, b, cv, d = cuts 101 return seq[:a] + seq[b:cv] + seq[a:b] + seq[cv:d] + seq[d:] 102 103 # --- Tabu Search --- 104 tenure = max(7, math.ceil(math.sqrt(n))) 105 tabu_expires = [0] * n # tabu_expires[txn] > it => txn is tabu at iteration it 106 s = list(best_s) 107 c = best_c 108 it = 0 109 stale = 0 110 MAX_STALE = max(50, 4 * n) 111 NBHD_BUDGET = 500 # max evaluations per neighborhood type per step 112 113 while not timed_out(0.97): 114 it += 1 115 best_nc = float(’inf’) 116 best_mv = None # (’sw’, i, j, ta, tb) or (’o1’, ti, pos, txn) 117 118 # Swap neighborhood: exhaustive if C(n,2)<=NBHD_BUDGET, else sample 119 if n * (n - 1) // 2 <= NBHD_BUDGET: 120 for i in range(n - 1): 121 for j in range(i + 1, n): 122 ta, tb = s[i], s[j] 123 tabu = tabu_expires[ta] > it or tabu_expires[tb] > it 124 s[i], s[j] = s[j], s[i]; nc = cost(s); s[i], s[j] = s[j], s[i] 125 if (not tabu or nc < best_c) and nc < best_nc: 126 best_nc, best_mv = nc, (’sw’, i, j, ta, tb) 127 else: 128 for _ in range(NBHD_BUDGET): 129 i = random.randint(0, n - 2) 130 j = random.randint(i + 1, n - 1) 131 ta, tb = s[i], s[j] 132 tabu = tabu_expires[ta] > it or tabu_expires[tb] > it 133 s[i], s[j] = s[j], s[i]; nc = cost(s); s[i], s[j] = s[j], s[i] 134 if (not tabu or nc < best_c) and nc < best_nc: 135 best_nc, best_mv = nc, (’sw’, i, j, ta, tb) 136 137 if timed_out(0.97): break 138 139 # Or-opt-1 neighborhood: exhaustive if n * (n-1)<=NBHD_BUDGET, else sample 140 if n * (n - 1) <= NBHD_BUDGET: 141 for ti in range(n): 142 txn = s[ti]; tabu = tabu_expires[txn] > it 143 rest = s[:ti] + s[ti + 1:] 144 for pos in range(n): 145 if pos == ti: continue 146 ns = rest[:pos] + [txn] + rest[pos:] 147 nc = cost(ns) 148 if (not tabu or nc < best_c) and nc < best_nc: 149 best_nc, best_mv = nc, (’o1’, ti, pos, txn) 150 else: 151 for _ in range(NBHD_BUDGET): 152 ti = random.randint(0, n - 1) 153 pos = random.randint(0, n - 1) 154 if pos == ti: continue 155 txn = s[ti]; tabu = tabu_expires[txn] > it 156 rest = s[:ti] + s[ti + 1:] 157 ns = rest[:pos] + [txn] + rest[pos:] 158 nc = cost(ns) 159 if (not tabu or nc < best_c) and nc < best_nc: 160 best_nc, best_mv = nc, (’o1’, ti, pos, txn) 161 162 if timed_out(0.97): break 163 164 # Apply best move 165 if best_mv is None: 166 # All moves tabu with no aspiration -> diversify 167 s = double_bridge(s); c = cost(s) 168 tabu_expires = [0] * n; stale += 1 169 else: 170 mt = best_mv[0] 171 if mt == ’sw’: 172 _, i, j, ta, tb = best_mv 173 s[i], s[j] = s[j], s[i]; c = best_nc 174 tabu_expires[ta] = it + tenure 175 tabu_expires[tb] = it + tenure 176 else: 177 _, ti, pos, txn = best_mv 178 rest = s[:ti] + s[ti + 1:] 179 s = rest[:pos] + [txn] + rest[pos:]; c = best_nc 180 tabu_expires[txn] = it + tenure 181 182 if c < best_c: 183 best_c, best_s = c, list(s); stale = 0 184 else: 185 stale += 1 186 if stale >= MAX_STALE: 187 # Restart with double-bridge from global best 89 188 s = double_bridge(list(best_s)); c = cost(s) 189 tabu_expires = [0] * n; stale = 0 190 191 # Final swap LS cleanup in remaining time 192 def swap_ls_final(seq, sc): 193 sv = seq[:] 194 imp = True 195 while imp and not timed_out(0.995): 196 imp = False 197 for i in range(n - 1): 198 for j in range(i + 1, n): 199 sv[i], sv[j] = sv[j], sv[i]; nc = cost(sv) 200 if nc < sc: sc = nc; imp = True; break 201 sv[i], sv[j] = sv[j], sv[i] 202 if imp: break 203 return sc, sv 204 205 fc, fs = swap_ls_final(best_s, best_c) 206 if fc < best_c: best_c, best_s = fc, fs 207 208 return best_c, best_s TXN: final evolved program 1 2 def get_best_schedule(workload, num_seqs): 3 """ 4 Beam search (B=20) construction -> SA with double-bridge reheating -> LS cleanup. 5 Beam search keeps top-B partial schedules at each step (vs single greedy path), 6 giving SA a better starting basin. SA with alternating double-bridge/from-best 7 reheating escapes local optima that greedy acceptance ILS cannot. 8 """ 9 import math 10 import time 11 try: 12 import numpy as np 13 from scipy.optimize import linear_sum_assignment 14 has_scipy = True 15 except ImportError: 16 has_scipy = False 17 18 n = workload.num_txns 19 start_time = time.time() 20 time_limit = 170 21 22 def timed_out(): 23 return time.time() >= start_time + time_limit 24 25 def double_bridge(seq): 26 if n < 4: 27 p = seq[:] 28 i, j = random.sample(range(n), 2) 29 p[i], p[j] = p[j], p[i] 30 return p 31 a, b, c = sorted(random.sample(range(1, n), 3)) 32 return seq[:a] + seq[b:c] + seq[a:b] + seq[c:] 33 34 # Phase 1: Build full pairwise cost matrix D[i][j]=get_opt_seq_cost([i,j]) 35 # Compute both D[i][j] and D[j][i] together; derive Copeland wins from same data. 36 # AP uses actual cost magnitudes, not just win counts like Copeland. 37 BIG = 1e12 38 D = [[BIG] * n for _ in range(n)] 39 wins = [0] * n 40 for i in range(n): 41 for j in range(i + 1, n): 42 cij = workload.get_opt_seq_cost([i, j]) 43 cji = workload.get_opt_seq_cost([j, i]) 44 D[i][j] = cij 45 D[j][i] = cji 46 if cij <= cji: 47 wins[i] += 1 48 else: 49 wins[j] += 1 50 copeland_order = sorted(range(n), key=lambda x: -wins[x]) 51 52 # Phase 2: AP relaxation -> initial permutation via linear sum assignment 53 # AP finds optimal transaction-successor assignment under pairwise cost model. 54 # Assignment creates disjoint directed cycles; follow them to build permutation. 55 if has_scipy and n >= 2: 56 D_np = np.array(D, dtype=float) 57 _, col_ind = linear_sum_assignment(D_np) 58 visited = [False] * n 59 chains = [] 60 for start in range(n): 61 if visited[start]: 62 continue 63 chain = [] 64 node = start 90 65 while not visited[node]: 66 visited[node] = True 67 chain.append(node) 68 node = int(col_ind[node]) 69 chains.append(chain) 70 # Stitch multiple chains via greedy best-cost insertion (both orientations) 71 if len(chains) == 1: 72 ap_seq = chains[0][:] 73 else: 74 chains.sort(key=len, reverse=True) 75 merged = list(chains[0]) 76 for chain in chains[1:]: 77 best_cost = float(’inf’) 78 best_merged = merged + chain 79 for fwd in [chain, chain[::-1]]: 80 for pos in range(len(merged) + 1): 81 trial = merged[:pos] + fwd + merged[pos:] 82 cost = workload.get_opt_seq_cost(trial) 83 if cost < best_cost: 84 best_cost = cost 85 best_merged = trial[:] 86 merged = best_merged 87 ap_seq = merged[:] 88 ap_cost = workload.get_opt_seq_cost(ap_seq) 89 else: 90 ap_seq = copeland_order[:] 91 ap_cost = workload.get_opt_seq_cost(ap_seq) 92 93 # Phase 3: Greedy from top Copeland seed (fallback comparison) 94 s_g = [copeland_order[0]] 95 rem_g = [x for x in range(n) if x != copeland_order[0]] 96 while rem_g: 97 bc, bt = float(’inf’), rem_g[0] 98 for t in rem_g: 99 nc = workload.get_opt_seq_cost(s_g + [t]) 100 if nc < bc: 101 bc, bt = nc, t 102 s_g.append(bt) 103 rem_g.remove(bt) 104 greedy_cost = workload.get_opt_seq_cost(s_g) 105 106 if ap_cost <= greedy_cost: 107 best_s = ap_seq[:] 108 best_c = ap_cost 109 else: 110 best_s = s_g[:] 111 best_c = greedy_cost 112 113 # Phase 4: SA with alternating double-bridge/from-best reheating 114 s = list(best_s) 115 c = best_c 116 sample_deltas = [] 117 for _ in range(min(80, n * (n - 1) // 2)): 118 i, j = random.sample(range(n), 2) 119 s[i], s[j] = s[j], s[i] 120 nc = workload.get_opt_seq_cost(s) 121 d = abs(nc - c) 122 if d > 0: 123 sample_deltas.append(d) 124 s[i], s[j] = s[j], s[i] 125 mean_delta = sum(sample_deltas) / len(sample_deltas) if sample_deltas else 5.0 126 T_init = mean_delta / math.log(10.0 / 3.0) 127 T_min = 0.005 128 sa_start = time.time() 129 sa_end = start_time + time_limit * 0.88 130 sa_duration = max(sa_end - sa_start, 1.0) 131 last_improve_time = sa_start 132 reheat_count = 0 133 134 while time.time() < sa_end: 135 now = time.time() 136 frac = (now - sa_start) / sa_duration 137 T = T_init * (T_min / T_init) ** min(frac, 1.0) 138 139 if T < T_init * 0.08 and now - last_improve_time > 5.0 and reheat_count < 6: 140 reheat_count += 1 141 T = T_init * (0.4 ** reheat_count) 142 if reheat_count % 2 == 0: 143 s = double_bridge(best_s) 144 c = workload.get_opt_seq_cost(s) 145 else: 146 s = list(best_s) 147 c = best_c 148 last_improve_time = now 149 150 if T < T_init * 0.02 and c > best_c * 1.03: 151 s = list(best_s) 152 c = best_c 153 154 r = random.random() 155 if r < 0.40: 91 156 i, j = random.sample(range(n), 2) 157 s[i], s[j] = s[j], s[i] 158 nc = workload.get_opt_seq_cost(s) 159 delta = nc - c 160 if delta < 0 or random.random() < math.exp(-delta / max(T, 1e-10)): 161 c = nc 162 if nc < best_c: 163 best_c = nc; best_s = list(s) 164 last_improve_time = now 165 else: 166 s[i], s[j] = s[j], s[i] 167 elif r < 0.80: 168 ti = random.randint(0, n - 1) 169 txn = s[ti] 170 rest = s[:ti] + s[ti + 1:] 171 pos = random.randint(0, n - 2) 172 s_new = rest[:pos] + [txn] + rest[pos:] 173 nc = workload.get_opt_seq_cost(s_new) 174 delta = nc - c 175 if delta < 0 or random.random() < math.exp(-delta / max(T, 1e-10)): 176 s = s_new; c = nc 177 if nc < best_c: 178 best_c = nc; best_s = list(s) 179 last_improve_time = now 180 elif n >= 3: 181 ti = random.randint(0, n - 2) 182 seg = s[ti:ti + 2] 183 rest = s[:ti] + s[ti + 2:] 184 pos = random.randint(0, len(rest)) 185 s_new = rest[:pos] + seg + rest[pos:] 186 nc = workload.get_opt_seq_cost(s_new) 187 delta = nc - c 188 if delta < 0 or random.random() < math.exp(-delta / max(T, 1e-10)): 189 s = s_new; c = nc 190 if nc < best_c: 191 best_c = nc; best_s = list(s) 192 last_improve_time = now 193 194 # Phase 5: First-improvement LS cleanup 195 def fi_swap(seq, sc): 196 sv = seq[:] 197 improved = True 198 while improved and not timed_out(): 199 improved = False 200 for i in range(n - 1): 201 for j in range(i + 1, n): 202 sv[i], sv[j] = sv[j], sv[i] 203 nc = workload.get_opt_seq_cost(sv) 204 if nc < sc: 205 sc = nc; improved = True; break 206 else: 207 sv[i], sv[j] = sv[j], sv[i] 208 if improved: 209 break 210 return sc, sv 211 212 def fi_or_opt(seq, sc, k): 213 sv = seq[:] 214 improved = True 215 while improved and not timed_out(): 216 improved = False 217 for i in range(n - k + 1): 218 block = sv[i:i + k] 219 base = sv[:i] + sv[i + k:] 220 for j in range(n - k + 1): 221 if j == i: 222 continue 223 ns = base[:j] + block + base[j:] 224 nc = workload.get_opt_seq_cost(ns) 225 if nc < sc: 226 sc = nc; sv = ns; improved = True; break 227 if improved: 228 break 229 return sc, sv 230 231 s = list(best_s) 232 c = best_c 233 prev = float(’inf’) 234 while c < prev and not timed_out(): 235 prev = c 236 c, s = fi_swap(s, c) 237 if timed_out(): 238 break 239 for k in [1, 2, 3]: 240 if timed_out(): 241 break 242 c2, s2 = fi_or_opt(s, c, k) 243 if c2 < c: 244 c, s = c2, s2 245 if c < best_c: 246 best_c = c; best_s = s[:] 92 247 248 return best_c, best_s 93

Instant research discovery