Paper deep dive
Evolutionary Ensemble Search: Council-Guided Program Evolution with Persistent Memory
Juan P. Madrigal-Cianci, Eshan Chordia
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 9/17/2026, 3:25:54 AM
Summary
The paper introduces Evolutionary Ensemble Search (EES), a framework for constructing machine-learning procedures via expert-guided program evolution. EES utilizes a role-specialized council to generate search directions, an orchestrator to allocate tasks, and an evolutionary engine that performs code mutation, pipeline edits, and crossover. The system incorporates persistent memory, operator credit, and validation-gated ensembling. It was evaluated on MLE-bench Lite, achieving medal-threshold results on 86.36% of tasks across various modalities.
Entities (8)
Relation Signals (7)
Evolutionary Ensemble Search → achievedresulton → MLE-Bench Lite
confidence 95% · A public MLE-bench Lite development ledger records medal-threshold artifacts on 19 of 22 tasks (86.36%)
Evolutionary Ensemble Search → uses → Orchestrator
confidence 95% · An orchestrator allocates these directions to execution specialists and an evolutionary engine.
Evolutionary Ensemble Search → uses → Council
confidence 95% · A role-specialized council turns task evidence and experimental results into structured search directions.
Impulse AI → published → Evolutionary Ensemble Search
confidence 90% · Juan P. Madrigal-Cianci Eshan Chordia Impulse AI
Evolutionary Ensemble Search → similarto → MLE-STAR
confidence 80% · EES shares their execution-driven approach.
Evolutionary Ensemble Search → similarto → FunSearch
confidence 80% · EES applies this principle to complete ML procedures.
Evolutionary Ensemble Search → similarto → MARS
confidence 80% · EES shares their execution-driven approach.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evolutionary Ensemble Search (EES) constructs machine-learning procedures through expert-guided program evolution. A role-specialized council turns task evidence and experimental results into structured search directions. An orchestrator allocates these directions to execution specialists and an evolutionary engine. The engine selects measured parents, diagnoses their errors, and produces descendants through code mutation, structured pipeline edits, and crossover. Each child must execute and acquire its own validation evidence. Population archives retain useful alternatives, while compatible predictions compete in a validation-gated ensemble stage. Search adapts through parent-relative operator credit, session memory, and problem-indexed lessons retrieved across runs. We specify these mechanisms, distinguish their execution profiles, and define the contracts required to compare candidates as their computations change. A public MLE-bench Lite development ledger records medal-threshold artifacts on 19 of 22 tasks (86.36\%), with best outcomes of 11 gold, five silver, and three bronze. The procedures span text, images, tables, audio, scientific geometry, and deterministic transformations. The campaign includes grade feedback between runs, external-source routes, and mixed confirmation procedures; its aggregate is an achieved development result, not a blind autonomous-agent success rate. The report contributes a concrete architecture for cumulative executable search and a versioned account of its cross-modal development outcomes.
Tags
Links
- Source: https://arxiv.org/abs/2609.17590v1
- Canonical: https://arxiv.org/abs/2609.17590v1
Trouble viewing inline? Open PDF directly →
Full Text
52,385 characters extracted from source content.
Expand or collapse full text
Evolutionary Ensemble Search Council-Guided Program Evolution with Persistent Memory Juan P. Madrigal-Cianci Eshan Chordia Impulse AI juan@impulselabs.ai eshan@impulselabs.ai Technical Report — September 2026 Public results and reproduction records: github.com/impulse-ai/mlebench-medals. Abstract Evolutionary Ensemble Search (EES) constructs machine-learning procedures through expert-guided program evolution. A role-specialized council turns task evidence and experimental results into structured search directions. An orchestrator allocates these directions to execution specialists and an evolutionary engine. The engine selects measured parents, diagnoses their errors, and produces descendants through code mutation, structured pipeline edits, and crossover. Each child must execute and acquire its own validation evidence. Population archives retain useful alternatives, while compatible predictions compete in a validation-gated ensemble stage. Search adapts through parent-relative operator credit, session memory, and problem-indexed lessons retrieved across runs. We specify these mechanisms, distinguish their execution profiles, and define the contracts required to compare candidates as their computations change. A public MLE-bench Lite development ledger records medal-threshold artifacts on 19 of 22 tasks (86.36%), with best outcomes of 11 gold, five silver, and three bronze. The procedures span text, images, tables, audio, scientific geometry, and deterministic transformations. The campaign includes grade feedback between runs, external-source routes, and mixed confirmation procedures; its aggregate is an achieved development result, not a blind autonomous-agent success rate. The report contributes a concrete architecture for cumulative executable search and a versioned account of its cross-modal development outcomes. 1 Introduction Machine-learning engineering involves choices about the computation itself: which data to use, how to represent them, what procedure to fit, and how to construct the final prediction. Search-based agents address these choices through code generation, targeted refinement, and repeated execution [1, 2]. An effective search must also preserve alternatives, connect observed errors to subsequent edits, and retain experience that remains useful after the current experiment ends. EES addresses this problem with a council–orchestrator–evolution architecture. Council members examine the same experimental state through different roles: discovery, methodology, memory, modeling, and implementation review. They return complementary structured fields rather than vote on a final answer. The orchestrator turns their output into bounded experiments. Evolution then acts on executable candidate procedures: it selects parents using measured performance, mutates their code or pipeline structure, evaluates the children, and retains candidates for further reproduction or predictive combination. Mutation is the principal transition between evaluated procedures. A parent with systematic residual error may motivate a feature transformation; a train–validation gap may motivate a different regularizer; a weak component may be replaced while the rest of the program is inherited. These are testable interventions, not accepted recommendations. The child has no fitness until it executes under the comparison protocol. This separation between proposing a change and measuring its outcome allows language-model reasoning to guide search without determining the result. The architecture retains experience at three timescales. Operator rewards change the distribution of local mutations. Session memory carries findings and failed approaches into subsequent council reviews. A persistent library associates task fingerprints with lessons, strategy statistics, and adaptable candidates. These mechanisms implement operational self-learning: experience changes the search policy without updating the underlying language model’s weights. The contributions are: 1. A council-guided evolutionary architecture with explicit parent-conditioned mutation, crossover, screening, archive retention, and prediction-level recombination. 2. A formal account of candidate inheritance, validation comparability, operator credit, and cross-run memory, with the differences between EES execution profiles made explicit. 3. A versioned analysis of the MLE-bench Lite development campaign, including all task scores, confirmation identities, representative procedures, and the scope of reproducibility. Here, algorithmic discovery denotes search over executable procedures and their computational structure. Scientific novelty additionally requires comparison with prior methods. The Lite study measures achieved solution breadth and performance, not the contribution of each search component. 2 Related Work Machine-learning engineering agents. MLE-bench evaluates the completion of Kaggle-derived ML tasks using local graders and historical human medal thresholds [3]. MLE-STAR retrieves candidate approaches, uses ablations to identify influential code blocks, targets those blocks for refinement, and constructs ensembles [1]. MARS combines cost-constrained Monte Carlo tree search, modular implementation, and comparative reflective memory [2]. EES shares their execution-driven approach. Its organizing mechanism is the interaction between structured expert review, a persistent evolutionary population, parent-relative operator feedback, and cross-run retrieval. Evolutionary program search. Genetic programming searches executable structures through selection and variation [4]. FunSearch combines language-model program proposals with evaluation and an evolutionary program database [5]; AlphaEvolve extends evaluator-guided code evolution to mathematical and computational problems [6]. EES applies this principle to complete ML procedures. Its variation interface accepts both typed configuration changes and diagnostic code edits, while the evaluation interface retains the data, prediction, and validation semantics needed to compare heterogeneous candidates. AutoML and predictive ensembles. Pipeline search, meta-learning, and reuse of previously evaluated models are established AutoML techniques [7, 8]. Ensemble selection can exploit a library of heterogeneous predictors by optimizing their combination on validation data [9]. EES incorporates these capabilities rather than treating them as new in isolation. It also admits generated code, deterministic transformations, scientific feature construction, and relationship-based procedures. Predictive recombination is distinct from program crossover: it combines outputs of fitted candidates rather than creating a new training program from their components. Reflective memory. Reflexion uses feedback-derived textual memory to influence subsequent trials without updating model weights [10]. MARS derives comparative lessons from differences between search branches [2]. EES separates this reflective state from numerical operator credit and from task-indexed cross-run records. This distinction identifies where experience changes the search: the next council context, the next mutation distribution, or the initialization of a future task. 3 Search Objects and Measurement Contracts 3.1 Task and executable candidate A task is T=(Dtr,Dte,m,C,B)T=(D_tr,D_te,m,C,B), where DtrD_tr contains available training examples, DteD_te contains test inputs, m is the metric, C is the output contract, and B gives resource limits. The contract identifies sample rows, targets, shapes, and prediction semantics. Any additional data source has an explicit access status; source availability and permission are distinct from predictive usefulness. An executable candidate is x=(ux,zx),ux=(dx,fx,hx,ax,px,qx),x=(u_x,z_x), u_x=(d_x,f_x,h_x,a_x,p_x,q_x), (1) where dxd_x specifies data access and training-side partitioning, fxf_x the representation or feature program, hxh_x the model or algorithmic structure, axa_x the fitting procedure, pxp_x inference, and qxq_x post-processing or ensemble topology. Hyperparameters and random seeds are fields of these components. The record zxz_x stores ancestry, execution status, diagnostics, measurements, and artifact identities. A candidate can be materialized from a structured pipeline or from source code. The mutable specification is separate from the comparison protocol κ. That protocol defines evaluation rows, target semantics, metric computation, and the information available during fitting. EES may explore a different training-side resampling scheme, but its fitness is comparable to a parent’s only under a common κ. Changing the comparison split does not create a valid improvement by itself: both procedures must be reevaluated on a shared protocol before their scores are ranked together. 3.2 Execution, eligibility, and cost The execution interface maps a candidate specification to predictions and a result record: Evalκ(ux,D,Bx)⟶(Y^xval,Y^xte,Qκ(x),zx).Eval_κ(u_x;D,B_x) ( Y_x^val, Y_x^te,Q_κ(x),z_x). (2) D denotes the admitted inputs, BxB_x is the allocated execution limit, and QκQ_κ orients the task metric so that larger values are better. The score is defined only when execution produces usable, comparable evidence. A timeout, exception, or invalid artifact is a typed failure, not an arbitrary low fitness. Prediction records carry row identifiers, target order, class semantics, and validation provenance. Let ℰgκE_g^κ be the eligible evaluated candidates available at generation g. For a nonempty pool, the incumbent is xg⋆∈argmaxx∈ℰgκQκ(x).x_g ∈ *arg\,max_x _g^κQ_κ(x). (3) A sample-shaped fallback can satisfy C without belonging to this validation-ranked pool. Equation (3) selects among completed experiments. Search consumes resources beyond training the selected candidate. For additive units such as device-hours, token expenditure, or monetary cost, Csearch=Cprofile/probe+Cproposals+Cexecution+Censemble+Ccontrol/memory.C_search=C_profile/probe+C_proposals+C_execution+C_ensemble+C_control/memory. (4) Failed work is included. Parallel resource use is distinct from elapsed wall time. Job deadlines and caps on generations, population size, parents, and offspring bound execution. 3.3 Population state and implementation profiles The common search state is g=(Pg,Ag,Kg,πg,Rg),S_g=(P_g,A_g,K_g, _g,R_g), (5) where PgP_g is the active evaluated population, AgA_g an archive of reusable candidates, KgK_g session memory and retrieved context, πg _g the mutation-operator distribution, and RgR_g the remaining budget. An execution log is maintained separately from AgA_g: failed candidates remain in the log even when they are ineligible for the population or Pareto archive. EES provides three execution profiles (Table 1). They share candidate, evaluation, and lineage interfaces but do not use identical search policies. The council is the task-facing control layer; it can direct specialist work or submit an evolutionary job. Section 5.1 states which aspects the campaign record can substantiate. Table 1: EES execution profiles. The nested profile supplies the detailed adaptive search policy in Sections 4.2–4.4. Profile Search representation Selection and feedback Multi-island Generated training code in general, tree-ensemble, and neural populations. Within-island tournament selection, code mutation and crossover, and Pareto consolidation. Nested Structured pipelines and optional code candidates. Diagnostic interventions, fitness–offspring parent sampling, operator credit, archive reuse, and cross-run lessons. Benchmark Registered modality operators and bounded configuration variants. Comparable-validation frontier, hypothesis probes, prediction alignment, and terminal benchmark grading. 4 Council-Guided Evolutionary Search Figure 1 shows the control and execution interfaces. The council proposes and critiques directions; the orchestrator assigns work; variation creates child specifications; execution supplies the evidence used by selection. Task evidence and retrieved memory Expert council structured review Orchestrator and specialists Measured parents and archive Directed mutation and crossover Child execution and validation Eligibility and selection diagnostic directives and budgetsevaluated survivors and reusable candidates Operator credit and run distillation Aligned predictions and final ensemble Figure 1: Council-guided executable search. The inner loop selects measured parents, constructs edits, executes children, and updates the population. Numerical operator feedback and distilled run records influence subsequent proposals. Predictive ensembling occurs only after alignment and validation checks; the external benchmark grade is outside this loop. 4.1 Council, orchestration, and initial evidence At council round r, expert e receives task T, compact memory KrK_r, and recent measurements VrV_r. It emits an expert-specific record ue,r=Φe(T,Kr,Vr)u_e,r= _e(T,K_r,V_r). The runner executes experts concurrently and merges their complementary fields into Ur=ℳ(ue,r:e∈r),U_r=M (\u_e,r:e _r\ ), (6) where rC_r is the active role set. Table 2 lists the roles. A model recommendation can coexist with a leakage warning or a record that the approach has already failed. Selection remains governed by executable evaluations. The orchestrator consults the council before substantive execution and after material results. It assigns data profiling, approach research, domain review, implementation, training, inference, or synthesis to specialists. Council outputs remain advisory: specialists and registered tools perform the artifact-changing operations. Council rounds and evolutionary generations have different cadences; a review can redirect multiple subsequent generations. Table 2: Role-specialized council outputs. The first three roles are always active; the final two occupy rotating slots. Expert Contribution to the search state Discovery Driver Next experiment, urgency, executable proposal, and a low-cost opportunity. Methodology Advisor Problem formulation, methodological fit, confidence, and checkpoint advice. Memory Keeper Established findings, attempted approaches, failed routes, and unresolved questions. ML Strategist Model family, feature intervention, validation design, and convergence assessment. Code Reviewer Implementation defects, statistical errors, leakage risks, and corrective actions. Initialization inventories tables, media, archives, auxiliary scientific files, and the sample submission. The resulting profile records identifiers, targets, modality, metric direction, sample counts, and file topology. A hypothesis links an observation to a candidate route and specifies prerequisites, expected artifacts, promotion criteria, and probe and execution budgets. Probes test whether the route is applicable: geometry requires auxiliary-file coverage; relationship matching requires identifiable correspondences; provenance matching requires measured precision and coverage. A promoted route initializes a candidate or constrains a mutation. Retrieved lessons supply priors, not exemptions from the task’s evaluation and access requirements. 4.2 Directed mutation: from parent evidence to child program Mutation starts from an evaluated parent, not only from the task description. For parent x, the diagnostic record ηx _x can include validation residuals, errors by data segment, train–validation gaps, fold variability, feature gaps, and prediction correlations with other candidates. The meso-level critic interprets this record; an architect converts the diagnosis into an edit directive and, where appropriate, a bias over variation operators. For operator o, mutation consists of proposal and application: δ∼Go(⋅∣ux,ηx,Ur,Kg,T),ux′=μδ(ux).δ G_o(\,· u_x, _x,U_r,K_g,T), u_x = _δ(u_x). (7) GoG_o specifies the proposal mechanism and μδ _δ applies the resulting edit. In code mode, the language model receives the parent program, relevant diagnostics, and permitted edit scope, then returns changed code. In pipeline mode, δ inserts, removes, replaces, or modifies typed fields. Both modes create a child specification whose unchanged components are inherited. Reuse of fitted weights or checkpoints is operator-dependent and must be represented in that specification. The child record identifies the parent, generation, operator, directive, and actual edit. Its validation predictions, score, runtime, and failure status are produced by a new execution. An inherited program fragment does not carry inherited fitness. This distinction also applies to warm starts: a retrieved procedure must be measured on the new task before it can become a parent. Table 3: Variation in EES. Examples specify admissible operations, not reconstructed trajectories or measured improvements in the Lite campaign. Supported operations depend on the execution profile. Operation What changes What the child must establish Configuration mutation Seed, solver, model family, regularization, or training allocation. Performance of the changed configuration under comparable validation. Pipeline mutation Feature expression, encoding, interaction, preprocessing block, or inference rule. An executable composition with valid intermediate and output semantics. Diagnostic code mutation A code block targeted using residuals, generalization gaps, or component diagnostics. A functioning edited program and its own predictions, cost, and score. Program crossover Compatible components from multiple parents, or a new program conditioned on them. Validity and measured behavior of the combined procedure; parent scores are not combined as child fitness. Ensemble mutation Member addition, pruning, or a change in predictive topology. Aligned predictions and a measured comparison for adoption or reuse. The benchmark profile implements a narrower mutation neighborhood: configured seeds, text-solver portfolios, tabular model families, and media training allocations. For such a child, θ′=θ⊕δθ =θ _θ, where ⊕ overrides designated fields. The operator is executed again with θ′θ . The nested and multi-island profiles additionally support structural or code-level variation. These are implementations of a common transition, not interchangeable descriptions of every run. A fresh draft supplies a new approach without a designated parent. Mutation extends one parent. Crossover conditions a child on multiple parents or transfers their compatible components. A restart restores exploration after a plateau. A code proposal inspired by two programs is distinguishable from literal gene exchange, and both are distinguishable from averaging their predictions. Admission and reproductive success. After execution, EES checks the artifact and comparison protocol before admitting x′x to the evaluated pool. An admissible child need not outperform its parent: it may be selected for diversity, preserved as a quality–complexity tradeoff, or used in an ensemble. Reproductive success is a separate decision. Under a shared protocol, the observed improvement is Δκ(x′,x)=Qκ(x′)−Qκ(x). _κ(x ,x)=Q_κ(x )-Q_κ(x). (8) The sign of this quantity records one measured comparison. It is not a guarantee about other random seeds or unseen data. Failed and rejected edits remain in the execution history so their cost and subsequent treatment can be inspected. 4.3 Selection, archives, and nested scheduling The nested profile balances measured quality with use of different lineages. For eligible parent xix_i, its sampling rule is wi=sigmoid(λQi−Q¯max(sQ,scv,ε))11+ni,pi=wi∑jwj.w_i=sigmoid\! (λ Q_i- Q (s_Q,s_cv, ) ) 11+n_i, p_i= w_i _jw_j. (9) Here Qi=Qκ(xi)Q_i=Q_κ(x_i), Q¯ Q and sQs_Q summarize quality in the sampling pool, scvs_cv is the selector’s cross-validation variability scale, λ>0λ>0 controls quality pressure, and ε>0 >0 prevents a zero denominator. The offspring count nin_i discounts repeatedly used parents. This reproductive-diversity factor is separate from predictive diversity and scientific novelty. The benchmark profile instead retains a ranked frontier; the multi-island profile uses tournament selection within biased populations. Population survival can also use predictive diversity, cross-validation variability, execution cost, and program complexity as profile-specific diagnostics. The nested archive preserves non-dominated quality–complexity tradeoffs with a configured capacity. An archived candidate may therefore contribute to crossover after it leaves the active population. The complete trace and the Pareto archive serve different purposes: the trace records what happened, whereas the archive stores candidates eligible for reuse. For the nested profile, let gO_g contain proposed mutations, crossovers, and new drafts, and let gκV_g^κ contain their eligible evaluated offspring. With incumbent elites EgE_g protected, the update is Pg+1=Eg∪Surviveρ(Pg∪gκ),Ag+1=ParetoCapρ(Ag∪Pg∪gκ), splitP_g+1&=E_g _ρ(P_g _g^κ),\\ A_g+1&=ParetoCap_ρ(A_g∪ P_g _g^κ), split (10) where ρ specifies the survival and archive-capacity settings. Unevaluated proposals remain outside the quality-ranked population. Restarts replace active candidates without rewriting the execution history. Scheduling operates at three timescales. The macro-loop retrieves task-relevant experience and adapts warm starts. The meso-loop invokes diagnosis periodically or after plateau or high-variance signals, then issues directed mutations and reallocates strategy effort. The micro-loop evaluates, selects, varies, and records candidates. Reduced-fold screening can limit expensive full evaluations; configured diagnostic and restart candidates can bypass that screen. Screen scores remain associated with their own protocol and are not substituted for full-evaluation fitness. 4.4 Operator credit and persistent memory The nested profile assigns credit relative to a child’s own parent. Let bt∈0,1b_t∈\0,1\ denote whether an attempt passes the configured parent-improvement gate. The gate uses paired fold-level evidence for fully evaluated children; non-improving and screen-rejected attempts receive zero reward. Thus the statistic measures success under the entire proposal-and-evaluation policy, including its screening decisions, rather than an intrinsic success probability for an operator. For operator o, let ℐo(g)I_o(g) index its credited attempts through generation g and let No(g)=|ℐo(g)|N_o(g)=|I_o(g)|. The empirical mean and exported operator weights are μ^o(g) μ_o(g) =No(g)−1∑t∈ℐo(g)bt,No(g)>0,μprior,No(g)=0, = casesN_o(g)^-1\! _t _o(g)b_t,&N_o(g)>0,\\ _prior,&N_o(g)=0, cases (11) πg+1(o) _g+1(o) =maxεπ,μ^o(g)+ε∑j∈maxεπ,μ^j(g)+ε,επ>0,ε>0. = \ _π, μ_o(g)+ \ _j \ _π, μ_j(g)+ \, _π>0,\ >0. (12) O is the finite operator set and μprior _prior is an optimistic initialization for untried operators. The floor in Eq. (12) applies to unnormalized weights; it preserves positive support but does not imply πg+1(o)≥επ _g+1(o)≥ _π. These exported weights normalize empirical credit; upper-confidence-bound statistics maintained by the bandit are separate. The paired-fold gate is a search heuristic for awarding credit, not a confirmatory statistical test of a scientific claim. Cross-validation folds share training examples, and many adaptively selected children are compared; a naive independence argument is inappropriate [11, 12]. The gate definition and threshold belong in the run configuration. Session memory stores the data profile, hypotheses, approaches, validation results, failures, constraints, and incumbent. The Memory Keeper compresses this state for later council rounds. End-of-run distillation extracts a lesson ℓj _j containing productive and wasted strategies, selected model family, useful features, ensemble or code-mode outcomes, and source-access warnings. A task fingerprint ϕ(Tj)φ(T_j) indexes the persistent library: ℒj+1=ℒj∪(ϕ(Tj),ℓj),K0(T′)=Retrievek(ℒj+1,ϕ(T′)).L_j+1=L_j∪\(φ(T_j), _j)\, K_0(T )=Retrieve_k(L_j+1,φ(T )). (13) Similarity-weighted retrieval provides a bounded context and strategy priors for the next task; compatible candidates can also warm-start its population. A failed strategy can lower a future preference without making that strategy inadmissible. Retrieved procedures are reevaluated under the new task’s contracts; the effect of retrieval on transfer remains an evaluation question. 4.5 Predictive recombination and finalization The prediction registry retains out-of-fold or holdout arrays from eligible candidates with row, target, class, and protocol metadata. Combination requires those semantics to agree. Equal array shapes alone are insufficient: class labels cannot be averaged as probabilities, and reordered rows cannot be aligned by position. Profile-specific ensemble search supports weighted blending, stacking, diversity-based selection, and hill-climbing selection from candidate libraries [9]. For aligned prediction matrices Y1,…,YJY_1,…,Y_J, a convex blend is Yw=∑j=1JwjYj,wj≥0,∑jwj=1.Y_w= _j=1^Jw_jY_j, w_j≥ 0, _jw_j=1. (14) If the rows of every YjY_j are probability vectors, the rows of YwY_w remain nonnegative and sum to one. Accuracy and calibration are assessed separately. Other combination methods require their own fitting and output checks. The automatic finalizer compares a proposed ensemble e with the best single procedure on a common pooled validation basis. It adopts the ensemble only when Qκ(e)>Qκ(x⋆)+τens,τens>0.Q_κ(e)>Q_κ(x )+ _ens, _ens>0. (15) Otherwise the single incumbent is retained. The gate prevents unconditional ensemble adoption; it does not make ensemble selection data an independent test set. Repeated choice of candidates, weights, or meta-learners can overfit internal validation [12]. Final generalization requires evaluation outside that selection process. 4.6 Execution record and terminal grading Material transitions produce structured records: task observations, council outputs, routing decisions, probes, parentage, mutation or crossover, execution status, validation metadata, archive decisions, ensemble membership, and final artifact identity. In a mutation record, the directive and the actual code or configuration change are separate fields. The latter identifies what was executed; the former explains what the search intended to test. These records support reconstruction when the referenced programs, inputs, dependencies, and model settings are available. They identify the exact computation to which an outcome or subsequent mathematical analysis applies. Within the benchmark profile, fitness uses internal validation. Official grading is disabled by default and, when requested, follows terminal artifact selection. Development mode permits the resulting grade to inform later engineering runs; evaluation mode treats grading as terminal for a frozen run. Algorithm 1 summarizes the profile-dependent control flow. Section 5.1 describes campaign-level feedback. Algorithm 1 Council-guided Evolutionary Ensemble Search Inputs: Task T, comparison protocol κ, execution profile ρ, and run library ℒL. 1. Profile the task; retrieve compatible lessons, strategy priors, and warm-start specifications. 2. Run the council and merge proposals and warnings. Route specialist work and probe route prerequisites. 3. Execute initial candidates under their budgets. Admit eligible results to P0P_0; retain every execution record. 4. While the generation and resource limits permit: (a) Select parents using the profile’s rule. On a diagnostic trigger, obtain a critic analysis and edit directives. (b) Propose mutations, crossovers, or new drafts; record parentage, edit scope, and actual changes. (c) Screen where configured; fully execute promoted or bypass-eligible children. Keep each score tied to its protocol. (d) Admit valid children, select survivors and elites, and update the reusable archive. Log failures and consumed resources. (e) Update parent-relative operator credit where enabled; return material results to session memory and council review. 5. Search compatible predictions for an ensemble; apply the profile’s finalization rule. 6. Freeze the chosen procedure and artifact. Distill permitted run evidence into the library. Invoke the external grader only after final selection, when requested. Profile scope. The nested profile enables all adaptive steps. Multi-island and benchmark jobs use the options in Table 1. 5 MLE-bench Lite Development Study 5.1 Benchmark, evidence unit, and protocol MLE-bench contains 75 Kaggle-derived tasks with local scoring and historical medal thresholds [3]. Its Lite split contains 22 low-complexity tasks covering tabular prediction, text classification, sequence normalization, image classification and restoration, audio, and scientific regression. Each task requires a prediction file satisfying its submission contract. A benchmark-assigned medal threshold is not a medal awarded in a live Kaggle competition. The study uses results/lite22-three-run.json at a fixed public repository snapshot [13]. Each task has a strongest recorded score and medal, a method description, and three confirmation entries, denoted C1–C3. A confirmation can be a newly trained seed, a separate process execution, or an exact prediction-file replay. The columns are not uniformly independent repetitions of a complete search. Let btb_t be the strongest recorded medal class for task t. The achieved campaign fraction is Rcampaign=122∑t=122bt∈gold,silver,bronze.R_campaign= 122 _t=1^221\b_t∈\gold,silver,bronze\\. (16) This statistic summarizes the campaign’s recorded outcomes. It is not a pass@1 estimate for a frozen agent, and raw metric values are not averaged across tasks. Task grades were visible between development iterations and influenced subsequent engineering. The ledger consolidates campaign-specific procedures rather than a uniform, equal-budget experiment. It does not include every attempted candidate, a consolidated compute total, or an artifact-by-artifact activation record for council review, online credit, and cross-run retrieval. The public documentation also retains assisted GPU executions and an assisted leaf result; their scores and identities must be distinguished from later ledger entries. Consequently, the results support achieved procedure construction, not a campaign-wide claim of fully autonomous execution. Appendix C identifies the provenance distinctions needed for reproduction. 5.2 External sources and measurement scope Four reported routes explicitly use external sources or historical lookup: Pizza uses external target lookup, Dog Breed uses external image lookup, Spooky Author uses a Gutenberg corpus, and Right Whale includes historical lookup or exploit discovery [13, method fields]. These results are retained in the development record and flagged in the task table. Retrieval of a withheld target is not evidence of predictive generalization on that target. A permitted external feature source, a provenance match, and target retrieval must not be treated as interchangeable inputs. Numerical grading checks a prediction artifact, not the provenance of its construction. MLE-bench separately addresses rule violations and contamination [3, Section 2.3]. An unmodified grader therefore does not establish protocol compliance. Removing the four explicit flags would also not certify the remaining tasks; that requires a source and trajectory audit. The benchmark profile’s aerial-cactus task-identity branch is another reason not to interpret its development coverage as evidence of task-agnostic transfer. 5.3 Aggregate outcomes and confirmations The ledger records medal-threshold artifacts on 19 of 22 tasks (86.36%), with best outcomes of 11 gold, five silver, and three bronze. The tasks without confirmed medals are NYC Taxi Fare, RANZCR catheter-line classification, and SIIM–ISIC melanoma classification. Their score entries are null; the ledger does not distinguish below-threshold scores from other terminal outcomes. Table 4: Recorded campaign outcomes. The best-artifact column and each confirmation column are different record sets. Counts are recomputed from the task-level entries, not additional benchmark experiments. Record set Gold Silver Bronze No medal Any medal Best recorded 11 5 3 3 19/22 C1 10 4 5 3 19/22 C2 10 4 5 3 19/22 C3 9 6 4 3 19/22 All three confirmation columns retain a medal on the same 19 tasks, but medal levels differ (Table 4). APTOS has a best silver artifact and three bronze confirmations. NOMAD changes from gold in C1 to silver in C2–C3. Seven tasks have a best score distinct from every confirmation score at recorded precision. Best-artifact performance and confirmation performance answer different questions. The 57 medal-bearing confirmation records correspond to 46 distinct task–submission-hash pairs. Dog Breed, MLSP Birds, Pizza, and both text-normalization tasks each repeat one submission hash across C1–C3; APTOS has two distinct hashes; the other 13 medal tasks have three. Different files need not come from independent training, and identical files need not come from the same process. The zero dispersion of the three any-medal fractions is therefore not an uncertainty estimate for blind autonomous performance. Appendix B supplies every score, medal class, metric direction, and within-task hash count. 5.4 Representative executable procedures Table 5 classifies the selected procedures by representation, fitting, and inference. It describes recorded solutions rather than reconstructed search trajectories. Table 5: Representative procedures from the public solution notes [13]. The differences concern executable computation, not only estimator hyperparameters. Task Recorded procedure Best result NOMAD Crystal-geometry descriptors and multi-output ExtraTrees regression. 0.05373; gold Dirty Documents CPU background estimation and paired-image reconstruction with per-pixel serialization. 0.01919; silver Dogs vs. Cats Image fine-tuning and combination candidates; the best artifact is distinct from confirmation members. 0.00597; gold Spooky Author Gutenberg-based provenance features with a TF–IDF fallback. 0.12422; gold Pizza External target lookup combined with a learned text fallback. 1.00000; gold Normalization Training-derived token lookup and deterministic transformation rules for English and Russian. Bronze in both Scientific representation. For NOMAD, the geometry-aware procedure parses lattice vectors, atomic coordinates, and species. It derives cell volume, density, lattice norms, species counts, and pair-distance statistics, then fits a multi-output ExtraTrees regressor over k-fold splits to predict formation and bandgap energies [13, NOMAD solution notes]. The recorded RMSLE of 0.05373 is below the stored gold threshold of 0.05589. This route uses auxiliary scientific structure within the common operator interface; the ledger contains no matched no-geometry ablation. Output-driven computation. Dirty Documents requires one output per pixel rather than one class per image. Its CPU procedure recognizes the image–row–column submission structure, estimates page background with a large-kernel box blur, and uses paired dirty and clean images to reconstruct intensities [13, Dirty Documents solution notes]. The confirmation RMSE values are 0.01926, 0.01928, and 0.01919, all silver. This case illustrates why an ML-engineering search space must include inference and serialization procedures as well as model selection. Deterministic and retrieval-based inference. The text-normalization routes use deterministic transformations and training-derived lookup. Spooky Author adds an external corpus, while Pizza retrieves targets from an external source. They share an execution abstraction but have different input requirements and statistical interpretations, as described in Section 5.2. 6 Discussion and Limitations What the system contributes. EES joins expert deliberation to an evolutionary population through explicit executable transitions. Council review and diagnostic intervention determine where to search; mutation and crossover determine how a procedure changes; validation and archive rules determine what remains available; memory determines which experience influences the next decision. The architecture therefore specifies more than an operator inventory. Its reusable unit is a procedure together with the problem, ancestry, and measurements needed to assess it. What the study establishes. The Lite record documents substantial cross-modal solution construction and achieved threshold performance. It does not establish that the full council, nested evolution, and memory stack jointly produced every artifact. Nor does it estimate the marginal benefit of those mechanisms, a uniform cost advantage, or blind generalization to untouched tasks. The broad architecture and the historical outcome record are separate forms of evidence until per-run manifests link their configurations and artifacts. Adaptive evaluation. Mutation selection, reduced-fold screening, archive reuse, and ensemble fitting all act on finite validation data. Parent-relative rewards are operational credit, not causal attribution, and repeated adaptive selection can exploit validation noise [12]. Similarly, retrieved memory can transfer task-specific artifacts or source-access mistakes. Task identities, source provenance, comparison protocols, and the contents of retrieved lessons must remain visible when evaluating transfer. Component and transfer evaluation. A controlled study should use a shared base seed pool, fixed evaluation protocol, and matched budgets, disabling council review, diagnostic mutation, online credit, archive crossover, final ensembling, and cross-run retrieval in separate conditions. Extra memory-derived warm starts must be recorded as part of the retrieval treatment. Best-so-far quality, area under the improvement curve, full-evaluation count, failure cost, and terminal held-out performance would measure distinct effects. Transfer requires a task split fixed before library construction and no feedback from evaluation tasks into that library. These are evaluation priorities, not results of the present study. Scientific novelty and reuse. Structural change alone does not establish scientific novelty. A discovery claim must identify the new method, its problem family, the closest prior methods, and its mathematical or empirical support. Lineage identifies the procedure being analyzed; it does not prove correctness, convergence, or robustness. Scientific extensions therefore require domain-specific representations, evaluators, and analysis. 7 Conclusion Evolutionary Ensemble Search joins expert review, parent-conditioned mutation, population retention, predictive ensembling, and persistent memory. Its central operation is to edit a measured parent, execute the child under a comparable protocol, and use the result to guide subsequent search. Operator credit and task-indexed lessons carry this feedback within and across runs. The MLE-bench Lite development record documents medal-threshold artifacts for 19 of 22 tasks across markedly different procedures. Together, the architecture and case study provide an implemented approach to cumulative ML-procedure search with explicit generation, evaluation, and reuse interfaces. Availability. The pinned public repository provides the ledger, solution notes, selected artifacts, and reproduction instructions. The EES engine is proprietary and not included. Appendix C specifies the snapshot and distinguishes artifact checking from complete search reproduction. References [1] Jaehyun Nam, Jinsung Yoon, Jiefeng Chen, Jinwoo Shin, Sercan Ö. Arık, and Tomas Pfister. MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement. arXiv:2506.15692, 2025. [2] Jiefeng Chen, Bhavana Dalvi Mishra, Jaehyun Nam, Rui Meng, Tomas Pfister, and Jinsung Yoon. MARS: Modular Agent with Reflective Search for Automated AI Research. International Conference on Machine Learning, 2026. arXiv:2602.02660. [3] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv:2410.07095, 2024. [4] John R. Koza. Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, 1992. [5] Bernardino Romera-Paredes et al. Mathematical discoveries from program search with large language models. Nature, 625:468–475, 2024. [6] Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery. Technical white paper, arXiv:2506.13131, 2025. [7] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. Advances in Neural Information Processing Systems, 28, 2015. [8] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren, editors. Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019. [9] Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes. Ensemble selection from libraries of models. Proceedings of the 21st International Conference on Machine Learning, article 18, 2004. [10] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023. [11] Yoshua Bengio and Yves Grandvalet. No unbiased estimator of the variance of K-fold cross-validation. Journal of Machine Learning Research, 5:1089–1105, 2004. [12] Gavin C. Cawley and Nicola L. C. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11:2079–2107, 2010. [13] Impulse AI. MLE-bench Lite-22: Results Ledger, Solution Notes, and Reproduction Records. Public repository, 2026. Snapshot 51c39605da4e0347b416c44a6c13086200959d45. Appendix A Execution Settings and Benchmark Operators The method specifies the common interfaces and identifies profile-dependent behavior in Table 1. A search rerun additionally requires council and model versions; data and checkpoint identities; random seeds; population, parent, archive, and generation limits; mutation scopes; screening folds and promotion fractions; diagnostic and restart triggers; the parent-improvement gate; ensemble settings; and the retrieved-memory snapshot. The campaign ledger does not consolidate these settings for every task. No numerical defaults are inferred from its scores. A.1 Benchmark operator catalog The benchmark profile exposes 12 callable entry points. A single entry point may contain an estimator portfolio, and campaign-specific recipes can contain additional implementations. Table 6: Callable entry points in the benchmark profile. Family Entry point Role Text TF–IDF portfolio Word and character features with validation-ranked solvers. Text Embedding Dense representation and linear prediction. Text Provenance Corpus matching with a learned fallback. Tabular Portfolio Encoded features, model-family variants, and seeds. Image Embedding Pretrained visual representations with validation models. Image Generic media Resource-bounded image feature baseline. Image Aerial specialist Specialized route used by the disclosed task-identity branch. Audio Generic media Audio-derived features under the common contract. Scientific Geometry Auxiliary-structure parsing and geometry-aware features. Relationship Duplicate Train–test matching and label transfer. Relationship Metadata Filename and metadata relationship inspection. Fallback Sample baseline Sample-shaped output and diagnostic status. Mutation and prediction interfaces. Benchmark variants change the configured random state, text solver or portfolio, tabular model family, or media training allocation. The runtime records generation, parent identifiers, changed fields, and rationale. Prediction registration separately records validation kind, row identity, target order, output shape, and artifact locations. Recombination can average selected predictions or fit weights on compatible validation arrays. Its comparability checks are runtime contracts, not formal guarantees about model correctness or input admissibility. Artifact availability. The public solution notes distinguish selected campaign recipes from this operator inventory. For example, the Dirty Documents denoising notes describe a specialized CPU procedure, while the catalog identifies the benchmark core’s callable families. A reader should not infer a one-to-one correspondence between the 12 entry points and the 19 medal-bearing solutions. Interface properties. Two elementary consequences explain the contracts. First, Eq. (14) preserves probability-row normalization when its hypotheses hold. Second, retaining an incumbent and its unchanged score under an unchanged κ makes the best recorded validation score nondecreasing by set inclusion. Neither property supplies a convergence theorem for program search or a bound on generalization error. If a score is reevaluated or the protocol changes, the second statement no longer follows from retention alone. Appendix B Complete Lite Results All results below are retained at the precision in the pinned public ledger. C1–C3 refer to the mixed confirmation procedures described in Section 5.1. The full task identifiers and individual submission and evidence hashes are available in the machine-readable ledger. Table 7: Best recorded scores and all three confirmation scores, with their medal classes. Arrows indicate whether higher or lower is better. U is the number of distinct submission hashes within C1–C3. E marks an explicitly documented external-source or historical-lookup route; an empty flag is not a clean-data certification. Task Best C1 C2 C3 U Flag Aerial Cactus ↑ 1.00000 G 1.00000 G 1.00000 G 1.00000 G 3 APTOS 2019 ↑ 0.92020 S 0.91942 B 0.91942 B 0.91930 B 2 Dirty Documents ↓ 0.01919 S 0.01926 S 0.01928 S 0.01919 S 3 Insults ↑ 0.91118 G 0.90164 G 0.90219 G 0.90284 G 3 Dog Breed ↓ 0.02439 B 0.02439 B 0.02439 B 0.02439 B 1 E Dogs vs. Cats ↓ 0.00597 G 0.00920 G 0.00870 G 0.00974 G 3 Histopathology ↑ 0.99585 G 0.99585 G 0.99578 G 0.99580 G 3 Jigsaw ↑ 0.98750 G 0.98723 S 0.98750 G 0.98701 S 3 Leaf ↓ 0.00328 S 0.00671 S 0.00328 S 0.00470 S 3 MLSP Birds ↑ 0.93170 S 0.93170 S 0.93170 S 0.93170 S 1 NYC Taxi ↓ – – – – – NOMAD ↓ 0.05373 G 0.05479 G 0.05997 S 0.05993 S 3 Plant Pathology ↑ 0.98902 G 0.98364 G 0.98902 G 0.97976 G 3 Pizza ↑ 1.00000 G 1.00000 G 1.00000 G 1.00000 G 1 E RANZCR ↑ – – – – – SIIM–ISIC ↑ – – – – – Spooky Author ↓ 0.12422 G 0.12422 G 0.12426 G 0.12424 G 3 E Tabular Dec. 2021 ↑ 0.95996 G 0.95885 G 0.95911 G 0.95887 G 3 Tabular May 2022 ↑ 0.99822 S 0.99821 B 0.99818 B 0.99822 S 3 Normalization: EN ↑ 0.99125 B 0.99125 B 0.99125 B 0.99125 B 1 Normalization: RU ↑ 0.97915 B 0.97906 B 0.97906 B 0.97906 B 1 Right Whale ↑ 0.99256 G 0.99238 G 0.99230 G 0.99230 G 3 E G, S, and B denote gold, silver, and bronze. Reading the table. The best score can fall outside C1–C3. For example, NOMAD’s 0.05373 best score is distinct from its three listed confirmations, and APTOS’s best silver is not reproduced at silver in any of those columns. A dash indicates a null entry, not an imputed numerical loss. The hash count identifies distinct prediction-file identities, not independent model fits. Flagged external-source routes remain in the campaign aggregate; an unflagged row is not a certification that the task satisfies a closed-data or blind-evaluation protocol. Appendix C Evidence Snapshot and Reproduction C.1 Public evidence identity The evidence snapshot is the public repository commit 51c39605da4e0347b416c44a6c13086200959d45. The ledger path is results/lite22-three-run.json; its Git blob identifier is 2ae7f4f57eec32b506b82c9147d68264b98f2ac4. The reported SHA-256 identifiers are: Ledger 4d9c4d07e53e743fc3600da63d1f0ea63394c36f92ab1d5cd1c0c1d77d49a59 Evidence board 663b9e3a56a12d0c69ac0c547921332d8341ef46bd4812a55a2a9a22b2680ea Manifest c6d2ad86653719ab55beabb70e549ffdd9e6c674790484d6ce45af668cf38c1 The board and manifest identifiers are fields reported by the ledger. Printing a digest does not establish availability of its preimage or replace hashing the exact bytes. The evidence commit identifies the public packet, not the private engine revision. C.2 Artifact-specific execution provenance The repository’s GPU reproduction notes describe assisted histopathology, MLSP Birds, and Jigsaw artifacts with scores 0.99154, 0.93143, and 0.98663, respectively, and identify an assisted leaf result. The aggregate best scores for the first three tasks are 0.99585, 0.93170, and 0.98750. Their differing values do not establish that assistance was removed, nor do the earlier assistance labels automatically classify the later artifacts. Each execution claim needs the associated program, run configuration, intervention record, and terminal artifact identity. Some solution directories store an earlier artifact while their notes quote the aggregate ledger. A replay must match the exact submission and grade record to the claim, not merely match a directory name or rounded score. The Dirty Documents notes, for example, state that the large prediction file is not committed. Availability therefore differs by artifact. C.3 Three reproduction levels Ledger checking verifies schema, task identities, counts, confirmation fields, and the identity of available bytes. Artifact regrading requires the exact prediction file, the correct benchmark data, and grader version. Search reproduction additionally requires the relevant EES implementation, model and dependency versions, access permissions, compute, and the selected profile’s configuration and memory state. These levels should be reported separately. A minimal artifact regrading command is mlebench grade-sample <submission.csv> <competition-id>The command presupposes benchmark setup and data access; it does not reproduce the search. The public quickstart and verification runbook describe the surrounding procedure.