Paper deep dive
Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes
Jingjie Ning, Xiaochuan Li, Ji Zeng, Hao Kang, Chenyan Xiong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/8/2026, 10:25:51 AM
Summary
This paper introduces 'Auto Research', a closed empirical loop driven by external measurement where specialist agents autonomously propose hypotheses, edit code, submit experiments, and use feedback to iteratively improve training recipes. The method operates across three environments (Parameter Golf, NanoChat-D12, CIFAR-10 Airbench96) without human intervention, producing an auditable trajectory of proposals, code diffs, and measured outcomes. It demonstrates significant improvements in validation loss, model quality, and training speed by leveraging shared lineage and role-partitioned search.
Entities (8)
Relation Signals (8)
Auto Research → developedby → Carnegie Mellon University
confidence 98% · School of Computer Science, Carnegie Mellon University
Auto Research → implements → Submitted-Trial Loop
confidence 95% · The method pairs externally measured training-recipe environments with a submitted-trial feedback loop.
Auto Research → improves → Parameter Golf
confidence 95% · reduces Parameter Golf validation bpb by 0.81%
Auto Research → improves → NanoChat-D12
confidence 95% · raises NanoChat-D12 CORE by 38.7%
Auto Research → improves → CIFAR-10 Airbench96
confidence 95% · reduces CIFAR-10 Airbench96 wallclock by 4.59%
Auto Research → uses → Specialist Agents
confidence 95% · We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials.
Submitted-Trial Loop → generates → Shared Lineage
confidence 90% · Lineage is the cross-trial record of hypotheses, diffs, scores, runtimes, statuses, and crash summaries read before the next proposal.
Specialist Agents → partition → Recipe Surfaces
confidence 90% · Specialist roles partition the editable recipe surface by environment constraints.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We study auto research as a closed empirical loop driven by external measurement. Each submitted trial carries a hypothesis, an executable code edit, an evaluator-owned outcome, and feedback that shapes the next proposal. The output is not a generated paper or a single model checkpoint, but an auditable trajectory of proposals, code diffs, experiments, scores, and failure labels. We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials. The central empirical finding is that lineage feedback lets agents turn evaluator outcomes, including crashes, budget overruns, size failures, and accuracy-gate misses, into later program-level recipe edits rather than one-shot suggestions. Across 1,197 headline-run trials plus 600 Parameter Golf control trials after one-time setup and launch, humans did not choose proposals, edit recipes, override scores, or repair failed trials during the search. In the three headline runs, the same submitted-trial loop reduces Parameter Golf validation bpb by $0.81\%$, raises NanoChat-D12 CORE by $38.7\%$, and reduces CIFAR-10 Airbench96 wallclock by $4.59\%$, with each task measured by its own external evaluator and legality checks. The trace includes a strict architecture-domain audit of 157 headline-run submissions and program rewrites such as a NanoChat attention-kernel path change. Within this scope the loop autonomously writes code, submits experiments, absorbs feedback, applies and combines known techniques inside each environment, and improves public starting recipes.
Tags
Links
- Source: https://arxiv.org/abs/2605.05724v1
- Canonical: https://arxiv.org/abs/2605.05724v1
Trouble viewing inline? Open PDF directly →
Full Text
94,823 characters extracted from source content.
Expand or collapse full text
Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes Jingjie NingXiaochuan LiJi ZengHao KangChenyan Xiong School of Computer Science, Carnegie Mellon University jening, xiaochu4, jizeng, haok, cx@cs.cmu.edu § GitHub Repository Abstract We study auto research as a closed empirical loop driven by external measurement. Each submitted trial carries a hypothesis, an executable code edit, an evaluator- owned outcome, and feedback that shapes the next proposal. The output is not a generated paper or a single model checkpoint, but an auditable trajectory of proposals, code diffs, experiments, scores, and failure labels. We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials. The central empirical finding is that lineage feedback lets agents turn evaluator outcomes, including crashes, budget overruns, size failures, and accuracy- gate misses, into later program-level recipe edits rather than one-shot suggestions. Across 1,197 headline-run trials plus 600 Parameter Golf control trials after one- time setup and launch, humans did not choose proposals, edit recipes, override scores, or repair failed trials during the search. In the three headline runs, the same submitted-trial loop reduces Parameter Golf validation bpb by0.81%, raises NanoChat-D12 CORE by38.7%, and reduces CIFAR-10 Airbench96 wallclock by4.59%, with each task measured by its own external evaluator and legality checks. The trace includes a strict architecture-domain audit of 157 headline-run submissions and program rewrites such as a NanoChat attention-kernel path change. Within this scope the loop autonomously writes code, submits experiments, absorbs feedback, applies and combines known techniques inside each environment, and improves public starting recipes. 1 Introduction Machine learning research advances by measured iteration: change code, launch experiments, read results, and choose the next move. This paper hands that propose-measure-revise loop to language agents under the same measurement environment a human researcher would use. Here, auto research means agents propose hypotheses, edit code, submit experiments, read evaluator-owned outcomes, and use them to revise later proposals. After one-time setup and launch, humans do not choose trials during search. Its unit is a submitted trial rather than a generated narrative: a hypothesis, executable code edit, evaluator-owned outcome, and feedback signal. The channel records successes and failures as measured evidence rather than polished summaries. Training recipes are a natural testbed because they expose architecture, data, optimization, schedules, losses, compression, and systems under constraints. An edit can improve quality but exceed a size cap, save time but miss an accuracy gate, or expose a bottleneck convertible into training tokens. These feedback shapes make the loop follow measured evidence rather than a fixed grid. Lineage is the cross-trial record of hypotheses, diffs, scores, runtimes, statuses, and crash summaries read before the next proposal. Specialist roles partition the recipe surface, while shared lineage carries measured evidence across roles so neighboring surfaces can build on it. Preprint. arXiv:2605.05724v1 [cs.MA] 7 May 2026 Figure 1: Closed-loop auto research trajectory. Submitted trials connect proposals, executable edits, external measurements, feedback, and the next research move. Prior work establishes pieces of this picture across repository-editing agents, machine-learning experiment agents, and evaluator-driven discovery systems. We study the empirical regime where these pieces form a sustained feedback loop over real training recipes, with executable edits, external measurements, failures, and follow-up proposals analyzed as one measured artifact. We study three environments with complementary feedback. Parameter Golf exposes size and budget pressure under a fixed FineWeb loss task [OpenAI, 2025, Penedo et al., 2024]; NanoChat-D12 exposes wallclock headroom and runtime bottlenecks in fixed-budget pretraining [Karpathy, 2025, Li et al., 2024]; and CIFAR-10 Airbench96 exposes an accuracy gate around speed improvements [Jordan, 2024, Krizhevsky, 2009]. Across the headline runs, the same loop improves all three starting recipes, and the traces expose auto research through code edits, launched runs, measurements, crashes, and follow-up proposals. The empirical object is the trajectory after successful and failed trials. The loop writes code, launches experiments, reads evaluator-owned outcomes, and uses feedback to revise later proposals. It improves public starting recipes, applies known techniques, and runs without human intervention during search. In these trials, agents combine and transfer known techniques rather than propose anything as structurally novel as the original Transformer. The contributions are to formulate auto research as an auditable closed-loop trajectory rather than a single generated output, instantiate it in compute-budgeted training-recipe development, demonstrate autonomous externally measured research without human intervention inside the search loop, and analyze measured lineage, program-level edits, failure feedback, evaluator-owned measurement, and role-partitioned recipe search. In a representative NanoChat-D12 trace, a systems agent diagnosed an attention-backend bottleneck. Recovered wallclock returned through lineage as budget headroom, later proposals spent it on more tokens, and the improved CORE score became the next current best. Figure 1 summarizes how proposals, code edits, external measurements, and lineage feedback become the next research move. 2 Related Work Evaluator-driven program search and parameter optimization. AlphaDev, FunSearch, and AutoML-Zero propose programs and let an evaluator decide validity [Mankowitz et al., 2023, Romera- Paredes et al., 2024, Real et al., 2020]. AlphaEvolve extends this to an evolutionary coding agent under automated evaluator feedback, but still targets algorithms and infrastructure rather than full training recipes [Novikov et al., 2025]. Hyperparameter optimization, population based training, and neural architecture search also use measured selection, usually over fixed parameter or architecture spaces [Bergstra and Bengio, 2012, Snoek et al., 2012, Li et al., 2018, Jaderberg et al., 2017, Zoph and Le, 2017, Real et al., 2019, Liu et al., 2019]. We keep the evaluator-driven pattern and move it to full Python training pipelines with data loading, optimizer state, schedules, kernels, evaluation, 2 and legality checks, where crashes, artifact caps, and runtime bottlenecks become feedback and the measured trajectory is analyzed, not only the final score. Language agents for code, machine learning, and long-running tasks. SWE-bench and SWE- agent test repository editing and agent-computer interfaces [Jimenez et al., 2024, Yang et al., 2024], while MLAgentBench and MLE-bench move agents into repeated ML experiments [Huang et al., 2023, Chan et al., 2024]. RE-Bench evaluates open-ended ML research engineering against human experts [Wijk et al., 2025]; MLGym-Bench frames open-ended AI research as agent environments and finds gains often come from hyperparameters rather than new hypotheses, algorithms, or ar- chitectures [Nathani et al., 2025]; AIBuildAI studies hierarchical model-building agents on MLE- Bench [Zhang et al., 2026]; and PostTrainBench asks frontier agents to improve LLM post-training under bounded compute while exposing reward-hacking failures [Rank et al., 2026]. The AI Scientist adds idea generation and paper writing [Lu et al., 2024], and Anthropic reports on effective, multi- agent, and long-running coding agents provide practical context [Anthropic, 2024, 2025b,a]. Our bounded setting instead makes the output a measured trajectory of code edits on fixed training tasks, so the closed empirical loop itself is the object of study. Compute-budgeted training and efficient training tools.Compute-optimal training studies how model size, data, and compute scale [Hoffmann et al., 2022]; nanoGPT, nanochat, Parameter Golf, and CIFAR-10 Airbench make related tradeoffs runnable at smaller scale [Karpathy, 2023, 2025, OpenAI, 2025, Jordan, 2024]. Parameter Golf uses a FineWeb-derived slice with artifact and wallclock limits [Penedo et al., 2024], nanochat provides an end-to-end language-model pipeline with CORE- style evaluation from DataComp-LM [Li et al., 2024], and Airbench provides fast CIFAR-10 recipes with explicit accuracy and time targets [Krizhevsky, 2009]. Final recipes often reuse tools such as FlashAttention and GPTQ [Dao et al., 2022, Frantar et al., 2023]. These tasks are cheap enough for repeated calls but strict enough to reject shortcuts, testing whether agents can choose and combine known tools under budgets without humans selecting the next trial. 3 Closed-Loop Auto Research Methodology The method pairs externally measured training-recipe environments with a submitted-trial feedback loop. The environment fixes editable files, the scored metric, legal failures, and evaluator feedback. The loop turns that feedback into later hypotheses and code edits. The four levels are task feedback, submitted trials, shared lineage, and parallel iteration. 3.1 Task environments and feedback signals We use three environments because they expose different feedback through the same submitted-trial loop. Parameter Golf rewards lower validation bits per byte on a fixed FineWeb-derived task with a 16 MB artifact cap and a 10 minute budget on eight H100 GPUs [OpenAI, 2025, Penedo et al., 2024]. We use the public 1.0810 leaderboard score as the denominator, keeping the delta tied to the public target record. Each trial returns score, status, exact byte counts, and per-phase timing, so the dominant feedback is size and budget pressure around the current bpb frontier. NanoChat-D12 rewards higher CORE from a fixed d12 nanochat pretraining run [Karpathy, 2025, Li et al., 2024]. The starting point is one calibrated run of the unmodified upstream recipe at the pinned commit, reaching 0.1618 CORE in our GPU environment. Agents can edit the coordinator script and vendored nanochat Python tree, but trials cannot download during execution. Tokenizer files, pretraining shards, and the evaluation bundle are prepared before launch. The protected parser extracts CORE from the log, and the main feedback is wallclock headroom under the fixed budget, because faster code can spend recovered time on more tokens. CIFAR-10 Airbench96 rewards lower shell-measured wallclock time, but only when mean CIFAR-10 accuracy reaches at least 0.96 [Jordan, 2024, Krizhevsky, 2009]. The starting point is the unmodified Airbench96 recipe calibrated to 26.356 s under our ten-seed cold-process protocol. The recipe cannot report its own time: the run script writes timing sidecars, and the classifier reads them. The main feedback comes from the accuracy gate, where fast near-misses return timing plus accuracy rather than a generic crash, making the miss usable for the next proposal. In all three environments, the starting recipe is fixed before search and the editable recipe does not own the evaluator. For each 3 frozen run, the harness, prompt templates, static knowledge files, and specialist taxonomy are fixed before launch; no human intervention occurs during that reported trajectory. 3.2 Submitted-trial loop A trial is the unit of the empirical loop. The task fixes editable files, score field, legality checks, and submission path. An agent reads current lineage, proposes a hypothesis, implements it as executable code, and submits a trial. An external evaluator measures the run, assigns status, and appends score, timing, and failure information. The next agent receives this feedback and refines the next proposal. Each agent session is a bounded LLM-agent SDK call, not an always-running process. It receives a fresh lineage view at session start, may submit multiple trials when a result exposes a concrete follow-up edit, and terminates under a tool-turn cap. All scores are measured outside the editable recipe. Parameter Golf uses the official evaluation path. NanoChat-D12 uses a protected parser and evaluator-side classifier path, with edits audited for parser or evaluator touches. CIFAR uses shell-side timing and rejects trials that miss the accuracy gate. This prevents reward hacking such as printing a better score or reporting fake runtime. 3.3 Specialist roles and shared lineage Specialist roles partition the editable recipe surface by environment constraints. The taxonomy is chosen before each run and fixed during search. Section 4.3 compares this role decomposition against generic multi-agent and single-agent controls. Parameter Golf has a broad recipe surface under a hard artifact cap, so its ten specialists cover architecture, optimization, quantization, regularization, loss, evaluation, curriculum, tokenizer, test-time training, and meta search. NanoChat-D12 is fixed- budget pretraining, so its five specialists cover architecture, optimization, data, schedule, and systems. CIFAR-10 Airbench96 is an accuracy-gated speed task, so its five specialists cover architecture, optimization, augmentation, loss, and regularization. Each specialist sees the same metric but receives a different recipe-surface prompt. This role conditioning makes sessions attend to different surfaces rather than repeatedly editing the most salient knob. The run log stores hypothesis text, diff summary, score, status, timing, and crash reason. The prompt renderer selects a compact lineage slice for the next trial, including the current best row, specialist recent rows, and adjacent-specialist rows. This preserves the frontier and keeps failed directions visible without replaying the full transcript. This setup also makes the research process a releasable artifact. Each trial has a proposal summary, code-diff summary, measured score, status label, timing record, and failure summary when applicable. These traces do not rely on private model internals, so they can be released with the harness and final recipes for audit, reproduction, and follow-up analysis. The public code and artifact archive is available at https://github.com/cxcscmu/Auto-Research-Recipes. 3.4 Measurement, calibration, and affordable iteration When hardware or run protocol differs from a public number, calibration runs before search and is append-only. This preserves logs and avoids stale denominators for NanoChat-D12 and CIFAR-10 Airbench96. Affordable iteration is a condition for closed-loop research because outcomes must return quickly enough to shape later proposals within the same search horizon. Our environments meet this condition because expensive phases are capped or short, while parallel submissions, score parsing, status classification, and legality checks run outside the editable recipe. For environmente, write the continuous wallclock for one submitted trial as a run, evaluation, queue, and logging decomposition. WithNindependent submitters using one shared blackboard, the measured throughput is τ e = τ run e + τ eval e + τ queue e + τ log e ; T e (N ) = N η ∥,e τ e ; η ∥,e = T e (N ) N T e (1) ∈ (0, 1].(1) We estimate this on Parameter Golf with the same starting recipe, 600 second budgets, and continuous wallclock only, excluding human pauses. Over the matched first-200-trial window, the single- generalist variant clears 2.26 trials per hour. The ten-specialist role swarm clears 18.15 trials per hour, givingη ∥,PG ≈ 0.80against the ideal10×speedup. The ten generic agents clear 16.79 trials per hour 4 A) Parameter Golf Target: validation loss (lower better) 0450900 1.07000 1.07750 1.08500 Best: 1.07221 Best @ 750 B) NanoChat-D12 Target: CORE score (higher better) 0100200 0.1000 0.1715 0.2431 Best: 0.2244 Best @ 156 C) CIFAR-10 Airbench96 Target: training time (lower better) 050100 24.72s 26.36s 28.00s Best: 25.15s Best @ 70 Figure 2: Best-so-far score over submitted trial index. Points are valid measured trials only; ineligible trials are omitted. The bold line shows best-so-far. withη ∥,PG ≈ 0.74. Thus the role-versus-generic difference in Section 4 is mainly proposal diversity and boundary discipline, not raw throughput. Efficiency is below one because submitters share the GPU pool, cluster queue, and blackboard filelock. This throughput matters because feedback helps the next proposal only when enough outcomes arrive within the same search horizon. 4 Experiments The experiments treat end-of-run score as a prerequisite, not the only object. We test whether the loop runs autonomously while writing code, submitting experiments, and collecting feedback; whether it improves each environment; whether submitted proposals include program-level changes rather than only numeric knobs; how outcomes distribute across roles; and how Parameter Golf controls isolate organization and feedback memory. The three headline runs contain 1,197 submitted trials: 900 in Parameter Golf, 200 in NanoChat-D12, and 97 in CIFAR-10 Airbench96. The three additional Parameter Golf control runs in Section 4.3 add 600 independent trials from the same starting recipe; the role-swarm control row in Table 1 is the first 200-trial window of the 900-trial headline run and is not counted again. Two historical 91-trial traces are retained only for proposal-diversity audit and excluded from these totals. We operationalize the loop through the trial log. Each trial records the proposing role, edit domain, proposal and diff summaries, status, score delta when valid, failure type when invalid, and timing or crash metadata. This is the observed proposal surface, not the latent distribution over every considered idea. We analyze which code edits reached the evaluator and how feedback shaped the trajectory. 4.1 Main trajectories All relative changes in Table 1 use the search starting point, not every external reference. For NanoChat-D12, the calibrated upstream d12 recipe at 0.1618 CORE is both baseline and fixed search start, so the final 0.2244 gain uses only that denominator. For CIFAR-10 Airbench96, the 27.3000 s reference and 26.3560 s start are the same upstream recipe under different protocols, so agent improvement is computed from the calibrated start. Table 1 summarizes external references, starts, headline runs, and Parameter Golf controls. Table 2 gives one compact representative per environment showing the loop is not only scalar recipe tuning, with the fuller list in Table 8. We audit submitted trials whose specialist or domain is architecture. This conservative, reproducible rule gives 95 of 900 Parameter Golf trials, 42 of 200 NanoChat-D12 trials, and 20 of 97 CIFAR trials, or 157 of 1,197 headline-run trials (13.1%). The count includes crashes, discards, disqualifications, and valid improvements because it measures submitted ideas, not only final-best contributors. We use this13.1%as a strict lower-bound sanity check, not an estimate of the full non-scalar edit fraction, because systems, optimizer, and loss specialists sometimes rewrite executable structure, such as the NanoChat attention-kernel path. The rows give representative submitted transformations outside a fixed HPO space. Each submitted trial records a proposal, code edit, evaluator status, and feedback for later proposals. Across headline-run trials, the logs contain 45 keeps and 592 valid non-improvements, plus boundary feedback such as size blocks, budget overruns, crashes, and accuracy-gate disqualifications. These 5 Table 1: Main experimental summary. Reference rows give external numbers, starting-point rows give the fixed search recipe and denominator for relative change, and run rows report change against that start. For Parameter Golf, 1.2244 is OpenAI’s official naive task reference, while 1.0810 is the public starting recipe. Dashes mark rows without submitted trials. Scores are rounded to four decimals here; exact trace values appear where individual controls are discussed. EnvironmentRowScore Rel. vs start Trials Valid impr. Naive reference baseline1.2244– Public SOTA starting point1.0810– Role swarm, full run1.0722-0.81%90036 Parameter Golf val_bpb (lower better) Role swarm control1.0731-0.73%20016 Single generalist1.0754-0.52%20014 Generic-101.0745-0.60%20010 No lineage1.0774-0.33%2003 NanoChat-D12 CORE (higher better) Calibrated upstream start0.1618– Role swarm0.2244+38.7%2005 Upstream reported reference(same code) 27.3000 s– CIFAR-10 Airbench96 train_s (lower better) Calibrated upstream start26.3560 s– Role swarm25.1464 s-4.59%974 Table 2: Compact representative submitted program transformations. Rows include valid and failed trials because they summarize generated research ideas, not only final-best contributors. EnvironmentTrial id(s)Concrete architecture or program change Parameter Golf 245, 475, 538Recurrent residual scaling; separate RoPE/NoPE query gains; per-head data-dependent attention-output gate. NanoChat-D12 007 SSSL to L attention path; masked SDPA math layers moved to Flash SDPA. CIFAR-10 Airbench96 040/044/053, 059/062 Residual-preserved ConvGroup depth reductions; wider-shallower blocks under the accuracy gate. rows are not discarded attempts: the case studies show how size, runtime, and accuracy-gate feedback return as follow-up edits. Figure 2 shows best-so-far score over submitted trial index using only valid measured points, including valid improvements and non-improvements. Ineligible trials are excluded. Earlier harness-vintage 91-trial traces, not prefixes of the 900-trial headline run, are retained in Table 3 as historical proposal- diversity audits. Figure 3 is the primary Parameter Golf control because it shares the modern harness vintage and adds generic multi-agent and no-lineage controls. 4.2 Loop behavior across roles Role-level outcomes provide trace context, but the submitted idea stream is primary. Appendix G reports role profiles, allocation balance, and tool-use summaries. The main text focuses on whether role-partitioned search changes the proposal surface and whether shared lineage carries ideas across role boundaries. Proposal entropy and idea sharing.We audit the submitted idea stream directly. For each trial, we embed only recorded hypothesis text with TF-IDF, excluding role names, domains, scores, statuses, and implementation notes. We cluster proposals online: a proposal joins the nearest centroid if cosine similarity is at least 0.30, otherwise it starts a new cluster. The effective proposal count isexp(H), whereHis Shannon entropy over cluster sizes. This does not recover unsubmitted latent ideas, but measures how diverse evaluator-facing ideas were. 6 Table 3: Submitted-proposal entropy and information sharing in Parameter Golf controls. TraceRowsCtx. Eff. clusters Top cluster Near dup. Cross-ctx. parent Cross-ctx. keep Shared idea clusters and limits Role swarm + lineage20010 (22)134.83.5%2.0% 154/184 (83.7%) 10/12 (83.3%) 5 of 29 clusters, 19 rows Role swarm, no lineage20010 (27)121.72.5%2.0% 155/174 (89.1%) 0/1 (0.0%) 5 of 28 clusters, 19 rows; parent IDs lack rich lineage feedback Generic 10-agent20010 (22)41.112.0%1.5% 125/158 (79.1%) 7/9 (77.8%) 22 of 30 clusters, 135 rows Single generalist2001 (200)61.917.5%10.1%n/an/an/a Historical swarm9110 (11)74.13.3%0.0% 76/86 (88.4%) 4/4 (100.0%) 5 of 8 clusters, 13 rows Historical single911 (91)39.37.7%10.0%n/an/an/a Role swarm (full run)90010 (96)439.62.2%1.1% 781/895 (87.3%) 32/33 (97.0%) 31 of 134 clusters, 168 rows Notes. Clusters use hypothesis text only; effective clusters areexp(H)at cosine threshold 0.30. Top cluster is the largest row share. Contexts are role or agent partitions, with max rows in parentheses. Parent and keep fractions report cross-context edges. Keep denominators include successful improvements whose declared parent falls inside the audited window. In the no-lineage run, declared parent IDs remain supervisor bookkeeping and rebasing anchors; because agents receive no within-run prior-trial content beyond the current-best score line, these edges are ancestry rather than information transfer. Table 3 reports matched Parameter Golf controls and keeps the historical first-91 rows as a compact proposal-diversity audit. In the historical harness-vintage traces, the single generalist has 39.3 effective clusters and a 10.0% near-duplicate rate, while the specialist swarm has 74.1 effective clusters and 0.0% near duplicates under the same TF-IDF vocabulary. The contexts column records proposal partitions across role or agent contexts, with maximum rows per context in parentheses. 0100200 0.000 -0.004 -0.008 Role Swarm Single Gen. Generic-10 A) Agent organization 0100200 0.000 -0.004 -0.008 Role Swarm No Lineage B) Lineage feedback Figure 3: Parameter Golf controls over the first 200 trials. The y-axis is delta validation bpb, lower is better. Panel A compares agent organizations, and Panel B removes shared lineage. The same audit exposes idea sharing through lineage.In the historical 91-trial Parame- ter Golf swarm trace, 76 of 86 within-window parent edges cross role boundaries. Of the 7 keeps in that window, the 4 with within-window parents all build on another role’s row. In the matched 200-trial controls, the role-decomposed lineage swarm has 10 of 12 successful keep parent edges crossing contexts, while no-lineage collapses to 0 of 1. The generic 10-agent con- trol shows parallel contexts alone are not enough: it has 10 contexts and many cross-agent parent edges, but only 41.1 effective clusters because identical prompts concentrate the stream. 4.3 Feedback lineage and organization controls The strongest control is the lineage ablation. The proposal-entropy audit tests whether the stream is repeated sampling, and the paired Parameter Golf memory ablation removes shared lineage while keeping the same starting recipe, specialist split, submitted-trial budget, and current-best score line. Figure 3 summarizes the Parameter Golf controls. Panel A compares the role-decomposed swarm with two agent baselines. The role-decomposed lineage run reaches 1.073142 with 16 valid drops by 200 trials, while the 10-agent generic control reaches 1.074495 with 10 drops. The same-harness single-generalist control finds 14 drops and reaches 1.075384, but its stream is more concentrated: the largest cluster consumes 35 of 200 submissions, including 32 preflight crashes around polar- coefficient edits, versus 7 of 200 for the role swarm. Panel A therefore shows that role-partitioned lineage improves score and boundary discipline under the same budget. Panel B shows the sharpest feedback effect under a matched 200-trial window. With lineage, the loop finds 16 valid drops and reaches 1.073142 by exp_176. Without lineage, it finds 3 drops, reaches 1.077413 at exp_075, then runs 125 submitted trials without a new valid improvement. Lineage acts as active research state by preserving which measured heads remain useful, which edits failed, and 7 which budget boundaries remain. The no-lineage run hits the eval-budget cap on61.5%of trials versus19.0%with lineage. The current-best stack already sits close to the600s eval cap, and only the lineage prompt’s Recent Activity block carries that dynamic SOTA fact across sessions. The no-lineage tree collapses to 3 active parent heads versus 15 with lineage, and specialists contributing at least one keep fall from 8 of 10 to 2 of 10. The generic multi-agent control has 13 declared parent heads, so lineage still maintains multiple measured frontiers, but it hits the eval-budget cap on59.0%of trials and has only 41.1 effective clusters. The control separates two mechanisms: shared lineage recovers much of the improvement count and parent-tree breadth, while role partitioning improves boundary discipline and broadens the idea stream. Diversity is not required for any valid drops, but here the less diverse stream finds fewer drops, ends at worse bpb, and collides with the eval-budget boundary more often. 5 Discussion The final recipes make the loop’s scope concrete. They show what the agents developed, how feedback became the next edit, and where current agents stop. The boxes summarize each final approach against its starting recipe, with the trial sequence that produced it. Appendix J gives long-form pipeline descriptions with post hoc explanatory schematics, and Appendix I records lower-level failure and measurement-audit cases. The boxes use a few recipe-specific terms. Evaluation-time adaptation and T-only z-loss are legal score-first Parameter Golf updates. Separate RoPE/NoPE query gains and attention-output gates are compact attention changes. TheSSSLtoLrewrite and logit bias are NanoChat runtime and model edits. CIFAR warmup repair restores accuracy after shortening the run. Appendix J.1 gives definitions. Parameter Golf final recipe Final developed approach. The final recipe improvesval_bpb from 1.0810 to 1.072210, a 0.81% reduction. It combines le- gal score-first evaluation-time adaptation with compact structural changes: recurrent residual scaling, separate RoPE and NoPE query gains, a per-head attention-output gate, GPTQ calibration changes, and a T-only z-loss objective. Trace to final recipe. The key trace converts 587, a size-blocked z-loss idea at 1.072431 and 2,056 bytes over cap, into size-valid 596at 1.072251 and 15,995,930 bytes by recovering artifact head- room.746’s eval-side drop reaches 1.072246 and750’s optimizer refinement reaches 1.072210. The loop turns boundary failure into a valid direction. The Parameter Golf case converts a score-useful but artifact-ineligible idea into a valid one once feedback names the byte boundary. The size-blocked z-loss trial returned a measured score and exact byte excess, which led to a follow-up edit that recovered artifact headroom while keeping the same score-first evaluation-time objective. NanoChat-D12 final recipe Final developed approach. The final recipe improves CORE from 0.1618 to 0.2244, a 38.7% gain. This systems-aware pretraining recipe rewrites the attention path fromSSSLtoL, moves all 12 lay- ers onto Flash SDPA, spends recovered wallclock on a longer run with target data ratio 12 to 100 to 130, and adds a zero-initialized logit-bias path after lm_head. Trace to final recipe. The trace runs from007’s runtime diagnosis, changingSSSLtoLand moving all 12 layers onto Flash SDPA, to 020’s larger token budget and biggest NC jump, +0.0334 CORE. 024raises the data ratio,025lands the 0.2241 schedule plateau, and156adds the zero-initialized logit-bias path to reach 0.2244 CORE. A measured systems fact becomes the next pretraining idea. The NanoChat-D12 case shows more than final-score improvement. The attention-path rewrite returned recovered wallclock through lineage as usable headroom. Later proposals spent it on more tokens, then refined the same head with a smaller logit-bias path. CIFAR-10 Airbench96 final recipe Final developed approach. The final recipe reduces training time from 26.3560 s to 25.1464 s while satisfying the 0.96 accuracy gate, a 4.59% speedup. It skips most logging-only validation calls, shortens the horizon, increases learning-rate intensity, and repairs the accuracy margin with faster warmup. Trace to final recipe. The trace moves from007’s validation- overhead removal to008and030’s shorter schedules, then uses 060’s 25.1650 s near-miss to motivate070’s warmup repair. The evaluator rejects fast but inaccurate code, and the loop turns the near-miss into a valid speed recipe. The CIFAR case uses an accuracy-gate miss as a research signal rather than a discard. The evaluator rejects fast but inaccurate code, the near miss returns a measured accuracy deficit, and the next successful edit repairs it while preserving most speed gain. 8 Together, the cases show auto research as a closed empirical trajectory rather than a one-shot generated artifact. Across headline runs, traces record proposals, executable edits, evaluator outcomes, and follow-up ideas. Submitted ideas go beyond scalar tuning: agents modify attention paths, optimizer updates, loss functions, recurrence scaling, quantization, proxy training, and gate-aware speed recipes, with proposals including GQA K/V projection rewrites, Bigram Hash Embeddings, MTP-2 objectives, self-paced loss caching, residual-preserved ConvGroup depth changes, and the NanoChatSSSLtoL attention-path rewrite. Role partitioning assigns priors to recipe surfaces, while matched controls in Section 4 show shared lineage and role-partitioned search broaden the submitted idea stream. Scope and limits. The cases mark what the closed loop can and cannot do under Section 3’s conditions. The observed boundary is compositional, where agents combine, transfer, and repair known techniques under external feedback while respecting constraints during a multi-day run. The loop is best suited to settings where failures become trusted, compact feedback within a bounded trial budget, and less suited to questions whose evidence is subjective or not automatically verifiable. The recorded trials do not show paradigm-level architecture invention such as a replacement for the original Transformer. Future agents may cross this boundary, and the same evaluator-driven feedback loop remains the natural arena for measuring whether such ideas hold up under real training. Within these limits the loop turns auto research from a one-shot claim into a continuously measured object. Future work. Future work follows the same conditions. Other compute-budgeted environments such as image, speech, or reinforcement-learning recipes can use the loop when trials are affordable and externally verified. Longer runs over weeks of continuous GPU time may expose cross-role composition beyond the matched 200-trial windows, and Eq. 1 helps plan that compute. Stronger future agents may also propose paradigm-level ideas rather than only compositional ones; the same evaluator-driven loop can measure whether they hold up under real training data. Releasing traces further enables retrospective human-versus-agent comparisons on the same environments. 6 Conclusion We studied auto research as a closed empirical loop that turns ML research into an inspectable sequence of executable proposals, code edits, evaluator-owned measurements, failures, and follow-up ideas. After one-time setup and launch, specialist agents ran 1,197 headline-run trials plus 600 Parameter Golf control trials by writing code, submitting experiments, reading external feedback, and propagating measured facts through shared lineage without humans choosing proposals or repairing failures during search. Across Parameter Golf, NanoChat-D12, and CIFAR-10 Airbench96, the headline runs improved fixed compute-budgeted recipes by0.81%,38.7%, and4.59%relative to their starting points. These gains show externally verified progress on real training pipelines rather than plans, reports, or scalar sweeps. Parameter Golf controls identify lineage feedback as a key mechanism for turning measured outcomes into later program-level edits, while NanoChat-D12 and CIFAR-10 case traces show the same pattern under different constraints. In NanoChat-D12, the loop converted a systems fact into more training tokens and a small logit-bias refinement. In Parameter Golf, it turned a score-useful but artifact-ineligible z-loss result into an artifact-valid keep. In CIFAR-10 Airbench96, a measured gate miss led directly to the final warmup repair. Each move is the same feedback loop applied to a different environment. The main lesson is that closed-loop auto research is useful and measurable when the environment owns the metric, per-trial cost is bounded, and outcomes return quickly enough to affect later proposals. Specialist agents cover many recipe surfaces, and Equation(1)captures how parallel submission scales the loop in continuous time. Within what current language models can compose, the pattern is practical and auditable because shared lineage preserves successes and boundary failures, experiments produce real feedback, and an evaluator the recipe cannot rewrite decides which proposals count. This changes agentic ML research from a final answer into a reusable record of what was tried, why, what failed, what improved, and how the next proposal changed. The same feedback loop can make empirical research more scalable, inspectable, and powerful as models become more capable. 9 References Anthropic. Building effective agents.https://w.anthropic.com/engineering/building -effective-agents, 2024. Engineering blog, published Dec. 19, 2024. Anthropic. Effective harnesses for long-running agents.https://w.anthropic.com/engi neering/effective-harnesses-for-long-running-agents, 2025a. Engineering blog, published Nov. 26, 2025. Anthropic. How we built our multi-agent research system.https://w.anthropic.com/en gineering/multi-agent-research-system, 2025b. Engineering blog, published Jun. 13, 2025. James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(10):281–305, 2012. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. MLE- bench: Evaluating machine learning agents on machine learning engineering, 2024. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. arXiv:2205.14135. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations, 2023. arXiv:2210.17323. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, 2022. arXiv:2203.15556. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation, 2023. Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, and Koray Kavukcuoglu. Population based training of neural networks, 2017. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. arXiv:2310.06770. Keller Jordan. cifar10-airbench.https://github.com/KellerJordan/cifar10-airbench, 2024. GitHub repository. Andrej Karpathy. nanoGPT.https://github.com/karpathy/nanoGPT, 2023. GitHub repository. Andrej Karpathy. nanochat: The best ChatGPT that $100 can buy.https://github.com/karpa thy/nanochat, 2025. GitHub repository. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URLhttps://w.cs.toronto.edu/~kriz/learning-features-200 9-TR.pdf. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. DataComp-LM: In search of the next generation of training sets for language models, 2024. 10 Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185):1–52, 2018. Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. arXiv:1806.09055. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery, 2024. Daniel J. Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, et al. Faster sorting algorithms discovered using deep reinforcement learning. Nature, 618:257–263, 2023. doi: 10.1038/s41586 -023-06004-9. Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob N. Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. MLGym: A new framework and benchmark for advancing AI research agents, 2025. Alexander Novikov, Ngân V ̃ u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wag- ner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Push- meet Kohli, and Matej Balog. AlphaEvolve: A coding agent for scientific and algorithmic discovery, 2025. OpenAI. Openai model craft: Parameter golf.https://openai.com/index/parameter-golf/, 2025. Online challenge description. Guilherme Penedo, Hynek Kydlí ˇ cek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale, 2024. Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg, Karina Nguyen, Matthias Bethge, and Maksym Andriushchenko. Posttrainbench: Can LLM agents automate LLM post-training?, 2026. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In AAAI Conference on Artificial Intelligence, pages 4780–4789, 2019. doi: 10.1609/aaai.v33i01.33014780. Esteban Real, Chen Liang, David R. So, and Quoc V. Le. AutoML-zero: Evolving machine learning algorithms from scratch. In International Conference on Machine Learning, 2020. arXiv:2003.03384. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nature, 625:468–475, 2024. doi: 10.1038/s41586-023-06924-6. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, volume 25, 2012. Hjalmar Wijk, Tao Roa Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Joshua M. Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Jun Koba Sato, William Saunders, Maksym Taran, Ben West, and Elizabeth Barnes. RE-bench: Evaluating frontier AI r&d capabilities of language model agents against human experts. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 66772–66832. PMLR, 2025. URLhttps://proceedings.mlr.press/v267 /wijk25a.html. 11 John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, 2024. arXiv:2405.15793. Ruiyi Zhang, Peijia Qin, Qi Cao, Li Zhang, and Pengtao Xie. AIBuildAI: An AI agent for automati- cally building AI models, 2026. Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017. arXiv:1611.01578. 12 Contents of Appendix A Specialist prompt templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 A.1 Global rules (Parameter Golf, abridged) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 A.2 Per-domain preambles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 A.3 Anti-anchoring and crash feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.4 Per-iteration user message. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 B Tool catalogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 C Trial classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 D Run configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 D.1 Per-task swarm configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 D.2 Model routing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 D.3 Doer session defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 D.4 Termination rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 E Hardware and execution environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 F Starting-point calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 G Additional trace statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 G.1 Historical single-generalist comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 H No-lineage ablation definition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 IFinal recipe and additional trace details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 JDetailed final solutions and schematics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 J.1Recipe-specific term glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 J.2Parameter Golf final recipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 J.3NanoChat-D12 final recipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 J.4CIFAR-10 Airbench96 final recipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 K Broader impacts and asset licenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 L Releasable trace contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 13 Parameter Golf global rules (abridged). You are one specialist in a multi-agent auto-research swarm working on the Parameter Golf challenge. Your goal every session is to propose a concrete edit to train_gpt.py, validate it locally (syntax + size), submit via the submit_trial tool, and learn from the returned row. One submit is a complete session; a second submit is allowed only when the first row surfaces a clear, concrete next edit –- otherwise stop. Hard limits (enforced by the harness) – Submission size ≤ 16,000,000 bytes (code + packed model). Comments and docstrings are auto-stripped before the LZMA pack. – Train wall ≤ 600 s; eval wall ≤ 600 s. – Each call to submit_trial produces one TSV row. Multiple submits per session are allowed; each is independently recorded. Tool protocol – cwd is your workdir_<domain>/. train_gpt.py lives there. – Bash is OS-sandboxed: reads are unrestricted; writes are confined to cwd. Write is not in your allowed tools. – submit_trial is the only GPU-burning tool. Use syntax_check and size_project freely before submitting. – WebSearch / WebFetch are the primary research channel for non-trivial design questions; the local PR library is supplemental cross-reference only. – Typical edit sequence: Read train_gpt.py → Edit(old, new) → syntax_check → size_project → submit_trial. On-demand knowledge files ../knowledge/LESSONS.md carries task-local setup notes; read when relevant, otherwise ignore. Workflow each session 1. Read LEADERBOARD, KNOWLEDGE.md, and Recent Activity in the user message; identify the current best. 2. Decide: mutate from the best, or rebase onto a non-best snapshot via rebase_to. 3. Mutate train_gpt.py via Edit. 4. Call syntax_check and size_project; fix and retry on failure. 5. Call submit_trial with a one-sentence hypothesis and a signed expected_delta. 6. Reflect on the result. One submit is a complete session; repeat from step 1 only if the returned row points to a specific next edit. Figure4:ParameterGolfGLOBAL_RULES,abridged.Thefullsourceisin multi_agent_pg/agents/prompts.py. NanoChat-D12 and CIFAR have analogous global rules with task-specific limits (e.g. NC’s 90-minute pretraining cap, CIFAR’s 0.96 accuracy gate). A Specialist prompt templates Each specialist’s session begins with a system prompt assembled from three pieces, in this order: 1.Knowledge files. Static markdown documents under the task package’sknowledge/ directory, concatenated and pinned at the top of the system prompt so the Anthropic prompt cache can amortise them across sessions. The set is task-specific and fixed before the reported run starts. 2.Global rules. A task-level protocol shared by every specialist on that task. Defines hard limits, the tool protocol, and the per-session workflow. Figure 4 reproduces the abridged Parameter Golf version. 3.Domain preamble. A specialist-specific scope and edit-radius statement. Figures 5, 6, and 7 show three of the ten Parameter Golf preambles; the remaining seven and the NanoChat-D12 / CIFAR preambles follow the same structure (scope + non-scope + edit-radius guidance). The per-iteration user message is rendered fresh from the live blackboard at every session start. Figure 8 shows the full-lineage form. Figure 9 shows the no-lineage ablation form (Section H). A.1 Global rules (Parameter Golf, abridged) A.2 Per-domain preambles The remaining seven Parameter Golf preambles (quant,tok,t,curr,loss,reg, eval) follow the same shape and are reproduced verbatim in the source repository at multi_agent_pg/agents/prompts.py. NanoChat-D12 has five preambles (arch,opt,data, sched,sys) and CIFAR-10 Airbench96 has five (arch,opt,aug,loss,reg). Per-task preambles never refer to other tasks’ constraint regimes. 14 Parameter Golf Architecture domain preamble. You are the Architecture specialist. Your scope is the transformer block itself: attention variants (full, sliding, differential, MLA), recurrence modules (GLA/Mamba/RWKV-style SSMs), residual topology (parallel vs sequential, Pre-Norm vs DeepNorm), MLP variants (SwiGLU, GeGLU, gated MoE-lite), normalisation (RMSNorm, sub-LN), embedding schemes (tied, factored, RoPE/ALiBi/xPos). You do NOT own optimizer, loss, dataset, or quantization –- those are other specialists’ domains. Small architectural tweaks (layer count ±1, dim ±64) are fine, but prefer changes that cross a qualitative line (e.g. swap a block type, add/remove a residual) when the current best has already been small-tweaked to death. Edit radius: your domain’s historical wins come from structural changes –- block type swap, residual topology flip, norm placement, attention head grouping. A single scalar tweak to an existing module (init_std 0.005→0.008) is hparam noise, not architecture –- that belongs in opt or meta. If your draft hypothesis is “change one number”, you’re probably in the wrong domain; pivot to a qualitative edit. Figure 5: One of ten Parameter Golf domain preambles. Each preamble follows the same three-part shape: scope (what the role owns), non-scope (what the role does not touch), and edit radius (the level of intervention that historically produced wins for this role). Parameter Golf Optimizer domain preamble. You are the Optimizer specialist. Scope: optimizer algorithm (Muon variants, Lion, Shampoo, Sophia), learning-rate schedule (cosine, WSD, linear warmup, per-param decay), momentum and weight-decay coupling, gradient clipping, LAWA/EMA weight averaging, per-tensor LR scaling. You do NOT edit model architecture or the loss function. Most of the value here lives in matching schedule shape to the 600 s budget –- do not propose schedules that implicitly assume more or fewer steps than the current best trains for. Edit radius: your domain’s wins come from schedule-shape changes –- swap the schedule family (cosine → WSD, linear → triangle), introduce a new warmdown phase, apply a different optimizer family to one parameter group, couple momentum↔LR in a new way. Single-coefficient tweaks (muon_wd 0.095→0.110, adam_eps 1e-8→1e-9) rarely exceed Fisher-info noise at our ∆ scale unless they cross a qualitative threshold. Figure 6: The Parameter Golf Optimizer preamble. Compared with Architecture, the scope is narrower (schedules, weight decay, optimizer family) but the same scope/non-scope/edit-radius shape is preserved. A.3 Anti-anchoring and crash feedback Specialists can repeatedly return to high-salience edits, which collapses the proposal stream toward a small set of canonical moves. Each specialist receives a short banlist of patterns that earlier sessions in its own role have already tried and that failed or returned within noise. The banlist is rendered alongside the lineage slice so the same dead end is less likely to be re-explored. Crash handling uses the same prompt channel. When a submitted trial crashes, the lineage slice for the next trial carries the deepest exception line and the deepest training-script frame from the crash. This gives the next specialist an explicit failure mode rather than a generic crash status, which makes surface variants of the same failed proposal less likely. A.4 Per-iteration user message B Tool catalogue Each agent session is wired with a small set of MCP tools (Table 4) plus the SDK built-insRead/ Edit/Bash/Grep/Glob/WebSearch/WebFetch/ToolSearch/TodoWrite/Agent.Write is deliberately not in the allowed-tools list: the artifact packer ignores any file that is not the canonical recipe file, so sidecars created via Write would silently inflate the artifact toward the size cap. Specialists can use web search and task-local knowledge files. The local library is helpful for known moves. Web search is most useful when a task is newly forked or when a failure points to a runtime dependency. The two channels provide complementary context for proposing and diagnosing submitted trials. Three SDK hooks moderate the tool channel: 15 Parameter Golf Meta-Search (analyst) domain preamble. You are the Meta-Search analyst. Scope: hyperparameter sweeps across the recent kept trials –- LR multipliers, batch size, warmup ratio, weight-decay, init scale. You are an ANALYST: you mostly read results.tsv + KNOWLEDGE.md to find a narrow hyperparameter tweak that several prior trials missed. Keep each tweak small-radius –- hyperparameter moves, not structural changes. Edit radius: your domain IS “small-radius”, but the radius should be on axes with HIGH unexplored volume, not crowded knobs. Before proposing, slice results.tsv via Bash to check how many trials already touched your proposed knob –- if > 5 in your own domain’s recent window, that axis is crowded; find a different one. For fresh axes, WebSearch recent hyperparameter-tuning literature first; the PR library’s gaps.md is a secondary reference for what has been ruled out in the environment. Figure 7: The Meta-Search preamble explicitly frames the role as an analyst who reads existing results before proposing. This is the only role in Parameter Golf that is allowed to lean on the lineage as its primary input rather than as a check on a freshly proposed edit. Per-iteration user message — full lineage. # Session start –- 2026-04-22T01:36:09Z You are specialist arch. Your workdir is <workdir_arch>. Current best: exp_037 (val_bpb=1.07683). ## LEADERBOARD.md <top-N keep rows by score, in compact bullet form> ## KNOWLEDGE.md <curated tree of prior hypotheses + outcomes; agent-readable summary> ## Recent Activity (most recent 10) - exp_046 [arch, discard, val_bpb=1.0775, ∆=+0.0007] differential attention with cross-thread V residual - exp_045 [opt, keep, val_bpb=1.07650, ∆=−0.0003] Muon WSD warmdown 25%→30% - ... ## (optional) ∆ Saturation signal <emitted only when last 5 own-role trials all |∆| < 5× 10 −4 > ## Your workdir - train_gpt.py present, 41,823 bytes head: """train_gpt.py –- 1.0810 SOTA stack...""" ## Your task this session Propose an edit within the arch domain, validate it locally, and submit via submit_trial. A single submit is a complete session; a second submit is allowed only when the first row points to a concrete next edit. Figure 8:Schematic of the user message rendered for each specialist session un- der full lineage.Concrete content is drawn fromLEADERBOARD.md,KNOWLEDGE.md, andthemost-recenttenrowsofresults.tsv.Theexactassemblyisin multi_agent_core/agents/base.py:render_user_message. • block_bash_writes(PreToolUse, onBash): denies destructive shell verbs (rm,mv,cp, in-place sed, tar -c, file redirects, package install, process control). Bash is read-only in the swarm. • block_bash_blackboard(PreToolUse,onBash):denies reads of black- board files (tree.tsv,results.tsv,lineage_snapshots/,events.jsonl, best.json,supervisor_audit.jsonl,anything underblackboard/) when MAGENT_NO_LINEAGE=1. Pass-through otherwise. • cap_builtin_tool_output(PostToolUse, onBash/Grep/WebFetch/WebSearch): truncates oversized outputs at 16 KB with a recovery marker, bounding cache growth without breaking legitimate slicing. C Trial classification The harness assigns one of nine status values to every submitted trial. Classification is performed by a post-trial parser (multi_agent_<task>/tools/run_classify.py) that reads the combined preflight + train + pack log and emits a JSONL row. The blackboard then maps that coarse status to the per-task enum in Table 5. 16 Per-iteration user message — no-lineage ablation. # Session start –- 2026-05-01T04:21:37Z You are specialist arch. Your workdir is <workdir_arch>. Current best: exp_000 (val_bpb=1.081). ## Your workdir - train_gpt.py present, 41,823 bytes ## Your task this session Propose an edit within the arch domain, validate it locally, and submit via submit_trial. A single submit is a complete session; a second submit is allowed only when the returned row points to a concrete next edit. ## Lineage policy (this run) Within-run prior-trial logs are unavailable for this run. The LEADERBOARD / KNOWLEDGE / Recent Activity sections you may have seen in other runs are intentionally absent. The read_snapshot and diff_snapshots tools are also disabled. Do NOT attempt to query blackboard files via Bash (tree.tsv, results.tsv, lineage_snapshots/, events.jsonl, best.json, supervisor_audit.jsonl, anything under blackboard/) –- those reads are rejected at the harness level. Propose from your static setup priors in the system prompt, the current-best score above, your workdir state, and your own in-session reasoning. Figure 9: Schematic of the user message under the no-lineage ablation (MAGENT_NO_LINEAGE=1). Sections that read within-run prior-trial outcomes are omitted. The current-bestexp_idand score are preserved because the agent uses them to rootrebase_to(best, workdir)at session start. See Section H for the operational definition. D Run configuration D.1 Per-task swarm configuration Each task package ships aswarm_config.jsonthat is the single source of truth for two per-specialist knobs: model assignment and GPU-cluster priority. The supervisor logs the resolved values at startup (supervisor_audit.jsonl). D.2 Model routing The framework supports per-role model routing, because different roles may need different reasoning depth. Architecture proposals tend to require deeper combinatorial search, while optimization, augmentation, and schedule proposals tend to be more tactical. All reported frozen runs use Claude Opus 4.7 for every role; only GPU-cluster priority differs by role. The routing policy is declared in swarm_config.json and kept fixed during each frozen run. D.3 Doer session defaults AspecialistsessionisasingleClaudeSDKClientcall.Defaultsfrom multi_agent_core/agents/base.py:DoerConfig: • thinking_budget = 8000 tokens of extended thinking budget. • max_turns = 200per session (cap on tool-use turns; sized for multi-submit sessions with PR-library drill-down). • enable_web = True(WebSearch/WebFetchare forced into the SDK’s preload list because the default preset would leave them deferred). •Sandbox:bubblewrap-based withBashread-anywhere / write-only-to-cwd. On hosts where pivot-root is unavailable (LXC, nested containers), the sandbox is auto-disabled and the block_bash_writes hook becomes the primary write barrier. •Permission mode:bypassPermissions(autonomous; the allowed-tools list is the safety boundary, not interactive prompting). D.4 Termination rules The supervisor stops on the OR of two conditions (multi_agent_core/supervisor/termination.py): 17 Table 4: In-process MCP tools surfaced to specialists. Names are namespacedmcp__apg__<name> on the SDK tool channel; the model sees both name and JSON-schema description automatically. Source: multi_agent_core/tools/ and multi_agent_pg/tools/. ToolDescription (one-liner)Required arguments submit_trialSubmit the specialist’s current train_gpt.py to a real eight-H100 evaluation. Runs local syntax + size preflight first; failures are recorded without GPU time. Blocks until the job finishes, then writes a row to results.tsv. specialist, hypothesis, expected_delta, parent_exp syntax_check py_compile the editable file and report any SyntaxError without executing. Millisecond-scale; catches the most common edit mistake before a GPU trial. workdir size_projectRun the real lzma+base85 pack step locally and report the projected packed size against the 16 MB cap. Used before submit_trial to catch oversize edits without burning a job. workdir param_countStatic AST estimate of trainable parameter count: sums n.Linear / n.Embedding literal sizes. Fast (∼5 ms) but only catches gross structural changes. workdir read_snapshotFetch the snapshotted source of a past kept experiment to study what a sibling specialist actually wrote. Truncates to∼200 KB. Disabled under MAGENT_NO_LINEAGE=1. exp_id (optional path) rebase_toCopy a past experiment’s snapshotted source into the current workdir, overwriting whatever is there. Used to fork from a non-best parent. exp_id, workdir diff_snapshotsUnified diff of two past experiments’ snapshotted source. Truncates to∼300 lines / 8 KB. Disabled under MAGENT_NO_LINEAGE=1. exp_a, exp_b read_pr_libraryFetch entry N from the curated PR library: technique, specialist tag, risk tag, available file paths. Cross-reference for web-found ideas; not the primary research source. pr_number read_pr_sourceReturn the extracted source text from one file inside a PR library entry. pr_number, path • Wall-clock deadline: default 48 h, configurable via –deadline-hours. •No-improvement grace: default 4 h since the most recentkeep, configurable via –no-improvement-hours. The grace clock resets every time a new keep lands. Astop.flagfile is written underblackboard/when either condition fires; in-flight specialist coroutines exit at their nextshould_stop()check, with a 60 s grace before forced cancellation. SIGINT and SIGTERM trigger the same shutdown chain plus a best-effort stop sweep over registered GPU-cluster jobs. E Hardware and execution environment All trials run on an internal GPU cluster. Each Parameter Golf and NanoChat-D12 trial is a fresh eight- H100 worker; CIFAR-10 Airbench96 trials run inside a long-lived GPU worker, which preserves the pre-warmed CUDA context across cold-process seeds. Supervisor and dashboard processes run on a head node with local ext4 storage. The blackboard, workdirs, and event logs all live on head-node local storage; only the editable workdir is synchronized to the worker-visible shared filesystem at trial submission time, and only the trial’s stdout and artifact are synchronized back. The Anthropic SDK uses the bundledclaudeCLI binary, which performs HTTP-level retries on transient errors (rate limits, network) before any exception is surfaced to the supervisor’s session-level retry path. Compute accounting.The reported submitted-trial counts include valid improvements, valid non- improvements, and failed submitted trials, so the headline and control totals already include most 18 Table 5: Trial status taxonomy. Source rules:multi_agent_pg/tools/run_classify.py, multi_agent_core/harness/tracker.py, and per-task adapters. StatusTriggering condition baselineThe single seed row written by bootstrap_from_baseline; not a submitted agent trial. keepValid measured run AND strictly better than the prior best in the metric’s preferred direction. discardValid measured run AND not strictly better than the prior best. crashTrain phase exited non-zero AND no valid score parse, OR pack step failed but train ran. preflight_crashLocal syntax check failed OR GPU-cluster submission failed before any GPU time was burned. size_blockedSubmission size> 16 MB at preflight (smoke pack) or post-run pack. train_budget_overrunTrain phase exceeded the task train budget plus 5 s tolerance for step-atomic granularity. eval_budget_overrunEval phase exceeded 600 s, or the outer trial timeout fired before eval completed. No tolerance: eval is continuous wallclock. disqualified (CIFAR only)Mean accuracy across cold-process seeds fell below the strict 0.96 gate. train_s is blanked and the row cannot win regardless of speed. harness_abortBookkeeping-side failure (e.g. cluster scheduler bookkeeping loss); not a substantive signal. Quarantined from the prompt’s Recent Activity. Table 6: Resolved per-specialist model assignment and GPU-cluster priority for each task. TaskModel assignmentGPU-cluster priority Parameter GolfAll ten roles on Claude Opus 4.7.All ten roles at priority 10. NanoChat-D12All five roles on Claude Opus 4.7. arch / opt / data at 10; sched / sys at 9. CIFAR-10 Airbench96 All five on Claude Opus 4.7. arch / opt / aug at 10; loss / reg at 9. failed-run compute inside the frozen search loops. A conservative cap-derived upper bound for active accelerator time is 4,000 H100-hours for the 1,500 reported Parameter Golf submitted trials (900 headline plus 600 controls, eight H100s, at most 600 s train plus 600 s eval per trial) and 2,400 H100-hours for the 200 NanoChat-D12 headline trials (eight H100s, at most 90 minutes per trial). CIFAR-10 Airbench96 is much smaller: 97 submitted trials run on one long-lived GPU worker, and even counting ten cold-process seeds per trial at roughly the reported 25–27 s scale gives under 10 single-GPU-hours of active training. These bounds exclude queue idle time and are upper bounds rather than summed per-job telemetry; preflight failures that terminate before a worker is launched consume less than the cap. The full project used additional setup and preliminary compute outside these reported totals, including benchmark preparation, starting-point calibration, harness and prompt development, and historical audit traces. Those runs are not counted as headline or control trials because they are not part of the frozen reported search loops. F Starting-point calibration The Parameter Golf, NanoChat-D12, and CIFAR-10 Airbench96 starting points used in Table 1 were established as follows. Parameter Golf. The starting point is the public 1.0810 record published on the Parameter Golf leaderboard. The search uses the corresponding starting code path, but the denominator remains 19 Figure 10: Specialist role partitioning and search behavior across environments. The top row shows the fixed role split chosen before search, and the bottom row sketches the resulting search pattern under each constraint regime. the public score rather than a newly calibrated local baseline. The search delta is therefore reported against the public number. NanoChat-D12.The starting point (0.1618) is one full run of the unmodified upstream NanoChat- D12 recipe at the pinned commit, scored by parsing the CORE line from the training log. This calibrated upstream run is both the baseline and the fixed search starting point for the reported 38.7% improvement. Calibration is append-only (python -m multi_agent_nc.calibrate_baseline –score 0.1618 ) so a populated run log is not silently overwritten. Without local calibration, the public “recipe starting point” from upstream NanoChat would be a stale denominator because the score depends on hardware, runtime image, and offline-mode settings. CIFAR-10 Airbench96.The starting point (26.3560s) is a ten-seed cold-process aggregate of the unmodified Airbench96 recipe under the strict0.96accuracy gate. The upstream reported reference of27.3000s is the same upstream recipe measured under a different reporting protocol, not an agent result and not the denominator for the 4.59% improvement. The cold-process protocol re-imports the recipe in a fresh Python process per seed so transient compile state does not bias the wallclock measurement. The reported search trajectory and selected recipe use the same cold-process protocol, so the timing source and accuracy gate match the main-paper CIFAR result. G Additional trace statistics This section reports role-level trace statistics that support the main text but are secondary to the closed-loop trajectory evidence. Allocation balance.Parameter Golf assigned 84 to 96 trials per role with CV 0.049. NanoChat-D12 assigned 33 to 45 trials per role with CV 0.100. CIFAR-10 Airbench96 assigned 18 to 21 trials per role with CV 0.062. The main differences across tasks therefore come from the mix of valid improvements, valid non-improvements, and ineligible outcomes rather than from one role receiving most of the budget. Tool-use patterns provide context but do not explain the task differences by themselves. In Parameter Golf, validation calls were frequent across roles, with 2.39 to 2.96 calls per trial. In NanoChat-D12, optimization used web search most often, but the systems role produced most valid improvements. In CIFAR-10 Airbench96, web and validation use were comparatively uniform. The stronger signal is how each role interacts with the task constraint. 20 Specialist outcome profiles across tasks Valid improvementValid non-improvementIneligible Bars show outcome share within each role. Top labels show valid improvements. arch 4 opt 5 quant 5 reg 2 loss 4 eval 3 curr 2 tok 3 t 5 meta 3 A) Parameter Golf Broad contribution 0 50 100 n=900 arch 1 opt 0 data 0 sched 1 sys 3 B) NanoChat-D12 Systems led 0 50 100 n=200 arch 0 opt 2 aug 1 loss 0 reg 1 C) CIFAR-10 Airbench96 Gate dominated 0 50 100 n=97 Figure 11: Specialist outcome profiles across the three environments. Each stacked bar is normalized within one specialist’s submitted trials. Green marks valid improvements, gray marks valid non- improvements, and orange marks ineligible trials. Labels above bars give valid improvement counts. Table 7: Specialist contribution patterns by task. Allocation was balanced across roles, so the table reports valid improvements and the dominant valid or ineligible boundary. TaskSearch shapeValid improvementsDominant boundary Parameter GolfBroadAll 10 roles contributed. Opt., T, and quant. contributed 5 each. Size gate, especially architecture, quantization, and tokenizer. NanoChat-D12ConcentratedSystems produced 3. Schedule and architecture produced 1 each. Most non-systems roles produced valid non-improvements. CIFAR-10 Airbench96Gate dominatedOptimization produced 2. Augmentation and regularization produced 1 each. 81 of 97 trials missed the accuracy gate. G.1 Historical single-generalist comparison The historical Parameter Golf single-generalist trace is useful as an audit of submitted proposal diversity, but it is not the primary causal control. Under the common rule that any legal lower val_bpbcounts as a valid improvement, the first 91 single-generalist trials produced 3 valid drops and reached a best reduction of 0.00122 bpb. The original specialist swarm produced 7 valid drops in the same 91-trial window and reached a best reduction of 0.00406 bpb. Four of those 7 swarm drops have declared parents that also fall inside the 91-trial window, which is the denominator used for the historical swarm keep-edge fraction in Table 3. This comparison changes role decomposition, the number of concurrently active proposal threads, and harness vintage. The single-agent run also predates the later anti-anchoring prompt revisions. We therefore use Figure 3 as the primary Parameter Golf control in the main text. The historical trace remains in Table 3 because it is directly useful for auditing proposal diversity and context partitioning. H No-lineage ablation definition Theno-lineageablationclosesthreelineagefeedbackchannelsunderasingle MAGENT_NO_LINEAGE=1 environment switch (set via the supervisor’s –no-lineage CLI flag). 1.Per-iteration prompt rendering. The user message rendered at every session start is short- circuited: theLEADERBOARD.md/KNOWLEDGE.md/ Recent Activity / Saturation-warning sections are dropped. The current-bestexp_id+ score one-liner is preserved (the agent uses it to root rebase_to). Figure 9 shows the resulting form. 21 Table 8: Full representative submitted program transformations. Rows include valid and failed trials because they summarize generated research ideas, not only final-best contributors. EnvironmentTrial id(s)Concrete architecture or program change Parameter Golf 001, 030/188Value-residual attention thread; parameter-neutral SwiGLU MLP replacing the non-gated squared activation. 245, 475, 538Recurrent residual scaling; separate RoPE/NoPE query gains; per-head data-dependent attention-output gate. NanoChat-D12 007 SSSL to L attention path; masked SDPA math layers moved to Flash SDPA. 022/094, 031, 104, 109 GQA K/V projections; learnable U-net skip; Bigram Hash Embedding; MTP-2 objective. CIFAR-10 Airbench96 040/044/053, 059/062 Residual-preserved ConvGroup depth reductions; wider-shallower blocks under the accuracy gate. 078/081, 091/093, 090 Self-paced loss caching replacing the proxy model; proxy-architecture rewrites; stochastic depth on residual paths. 2.Lineage-reading tools.read_snapshotanddiff_snapshotsare removed from allowed_toolsandpreload_tools.rebase_tois preserved because it does not return prior-trial content to the agent — it copies code into the workdir using the already-known current-best exp_id. 3.Bash reads of blackboard files. Theblock_bash_blackboard PreToolUsehook (Section B) rejectsBashcommands matching any oftree.tsv,results.tsv, lineage_snapshots/,events.jsonl,best.json,supervisor_audit.jsonl, or any path underblackboard/. This closes the in-practice dominant residual: an empirical audit of the lineage-on Parameter Golf run found that57.9%ofBashcalls with parseable arguments targeted blackboard files (mainlyawkslices oftree.tsv); without this hook, the prompt-side ablation would leak heavily. What is intentionally not ablated: •Static priors. The system-prompt knowledge files and the global rules + domain preamble are kept. These are task setup priors fixed before the reported run starts; removing them would test “zero-shot” agent behaviour, not the value of within-run feedback memory. •Current-best score. The agent seesCurrent best: exp_<id> (val_bpb=...)as a one-line entry. Without this, the agent cannotrebase_toa usable starting code state and the loop is no longer comparable. •Workdir code state. Each specialist’sworkdir_<role>/train_gpt.pyis the agent’s edit target and necessarily reflects the current-best code. The agent canReadthis file. This is the ablation’s irreducible residual: an editable closed-loop process must give the agent something to edit. The ablation therefore tests whether agents can produce diverse, valid proposals when given (a) static priors, (b) the current-best code, and (c) their own in-session reasoning, but NOT (d) any record of within-run prior trials’ hypotheses, scores, statuses, diffs, or crash logs. I Final recipe and additional trace details The main text presents the final developed approaches at a high level. This appendix records the supporting trace details that are useful for audit but too fine-grained for the main case analysis. Failure rows as boundary evidence.Failed trials are not the main product, but they are part of the feedback loop. CIFAR ineligible rows map the speed and accuracy boundary. Parameter Golf size and evaluation failures name which proposal families are too expensive or too large. NanoChat-D12 has few real crashes because preflight catches common failure classes before a full run starts. A compact memory of failure type, crash excerpt, and phase timing gives the next agent a concrete boundary to respect rather than a vague instruction to try something different. 22 Table 9: Final-recipe components and trajectory evidence. Major components are code-level or protocol-level changes. Small coefficient adjustments are listed only when they are the final polish on a larger branch. TaskFinal improvementMajor developed componentsTrajectory evidence Parameter Golf1.0810 to 1.072210, 0.81% lower Score-first evaluation-time adaptation with T-only z-loss, loop-aware residual scaling for recurrent blocks, separate RoPE/NoPE query gains, attention-output gating, and GPTQ/Hessian calibration changes. The final optimizer edit decouples Muon warmdown cool target from warmup start. exp_587 finds useful z-loss but exceeds the size cap. exp_596 repairs the artifact and becomes legal. exp_746 and exp_750 refine the same head. NanoChat-D120.1618 to 0.2244, 38.7% higher Attention-path rewrite from SSSL to L so all layers use Flash SDPA in the local GPU environment, expanded training-token budget under the same 90 minute cap, and a final learnable logit-bias path after lm_head. exp_007 identifies the backend bottleneck. exp_020, exp_024, and exp_025 spend recovered wallclock on more tokens. exp_156 adds the vocabulary-prior path. CIFAR-10 Airbench96 26.3560 s to 25.1464 s, 4.59% lower Gate-aware speed recipe that skips most logging-only intermediate validation calls, shortens the training horizon, raises learning-rate intensity, and completes warmup earlier. Pure architecture and proxy-model speedups were tried but usually missed the 0.96 accuracy gate. exp_007 and exp_008 remove overhead and shorten the run. exp_030 reduces evaluation overhead further. exp_060 shows a fast near-gate miss. exp_070 repairs the gate with 5% warmup. Measurement turns proposals into evidence. The environment must own the metric. If the editable recipe can report its own time, accuracy, or loss, the agent can improve the row without improving the training run. This is why CIFAR uses shell-side timing, NanoChat-D12 parses the training log, and Parameter Golf uses the external evaluation path. Calibration serves the same purpose at the starting-point level. Hardware, runtime images, offline settings, and seed protocols change the absolute number. Running the unmodified recipe under the same protocol makes the relative improvement meaningful. Evaluator-touch audit.We audit the edit surface for evaluator contact. In NanoChat-D12, visible edits touch model, training, data, optimizer, and experiment files, with zero edits to sensitive evaluator or parser paths. In CIFAR-10 Airbench96, all visible edits touchairbench96.py, with zero edits to the timing classifier or result parser. Parameter Golf uses an older event schema, so its evaluator-touch audit relies on archived snapshot inspection rather than edit-stream paths alone. The audit supports measurement hardening as a design choice, not as an ablated mechanism. Concrete feedback repair cases.Three rows illustrate how measured feedback becomes the next edit. In Parameter Golf, exp_587 measuredval_bpb=1.072431 with T-only z-loss but missed the 16 MB artifact cap by 2,056 bytes. exp_596 retained the same z-loss mechanism and recovered byte headroom, turning the idea into a legal keep at 1.072251. In NanoChat-D12, exp_007 moved the attention path to Flash SDPA and exposed large runtime slack. exp_020 and exp_024 used that slack for more training tokens, then exp_025 converted the same direction into the main plateau. In CIFAR-10 Airbench96, exp_060 was fast at 25.1650 s but missed the gate with 0.959560 accuracy. exp_070 kept the 42-epoch, lr=11 speed recipe and changed warmup from 10% to 5%, reaching 25.1464 s with 0.960080 accuracy. J Detailed final solutions and schematics This appendix gives a full prose description of the final recipe on each environment, lays out each developed component against the inherited starting stack, and summarizes the data flow with a schematic. The schematics were produced post hoc as explanatory figures by Claude Design and 23 Figure 12: Parameter Golf final recipe schematic. The figure summarizes inherited and rewritten components, the score-first evaluation-time adaptation path, the feedback signal that re-enters lineage, and the final artifact. Post hoc Claude Design-generated explanatory schematic; not part of the search loop. were not part of the search loop itself. Components that the closed-loop search added or rewrote are marked in teal in every schematic. J.1 Recipe-specific term glossary Parameter Golf.Evaluation-time adaptation denotes score-first test-time updates inside the sliding- window evaluation flow: a chunk is scored under the current model before any gradient update from that chunk, and only then can that chunk train the model for later chunks. T-only z-loss adds the z-loss objective only during those score-first updates, so the auxiliary objective does not consume the main train budget. Separate RoPE and NoPE query gains apply different learnable scalar gains to rotary and non-rotary projection heads, and a per-head attention-output gate multiplies each head’s attention output by a data-dependent scalar before the residual add. NanoChat-D12.TheSSSLtoLattention path rewrites the 12 layer body from short masked-SDPA sliding-window layers mixed with longer Flash-SDPA layers into a uniform stack of long Flash-SDPA layers. Flash SDPA is the IO-aware attention kernel used by that dispatch path. The logit-bias path is a zero-initialized learnable vector added after lm_head and trained as a vocabulary-level prior. CIFAR-10 Airbench96.The warmup repair reduces the warmup ratio so the schedule reaches its peak earlier, recovering the accuracy margin lost when the run is shortened. J.2 Parameter Golf final recipe The starting recipe is the code path corresponding to the public 1.0810 Parameter Golf record. Its inherited stack is a SentencePiece-8192 BPE tokenizer, a transformer body with three layers carrying 24 a long-context recurrence path stacked on parallel residual sublayers, a squared-SiLU activation in the MLP, full attention with a single rotary positional embedding for all heads, a Muon optimizer on most parameter groups with AdamW on the embeddings and language-model head, a single warmup-stable-decay schedule, GPTQ post-training quantization with a hand-tuned percentile clip, and a two-stage submission packer that first applies Brotli-11 to the quantized weights and then wraps the resulting blob plus the source code in an lzma-plus-base85 self-extracting file under the 16 MB cap. The final 1.072210 recipe keeps the high-level structure of this stack and changes a small number of components in code. Inside attention the search adds separate query-gain scalars for the rotary and non-rotary projection heads, so that the same K-cache services both. It also adds a per-head data-dependent output gate, where the gate values are computed from the incoming residual features and used to scale each head’s attention output before the residual add. Inside the recurrence path, the looped block applies a fixed rescaling ofln_scale_factordivided by the square root of(num_loops + 1)to each pass, which keeps activation magnitudes stable as the loop count grows inside the same parameter budget. The MLP keeps the squared-activation pattern from the starting recipe rather than being rewritten into a gated SwiGLU. The optimizer keeps the Muon-plus-AdamW split but decouples the Muon warmdown cool target from the warmup start, so the schedule has independent control over the early and late phases. The most distinctive change happens at evaluation time. Inside the official sliding-window evaluation flow, the code scores each chunk under the current model before using that already scored chunk for a short test-time training update. The agent loop discovered that adding z-loss only to those score-first T updates, and not to the main training run, gave a measurable bpb drop without hitting the size cap. GPTQ is rewritten so that calibration uses a Full-Hessian path with SDClip percentile clipping rather than a fixed percentile, which changes the calibration path in a way that reduces quantization error at the same byte budget. The trace from587to596that turned the size-blocked z-loss idea into a valid keep recovered the necessary 2,056 bytes through source-side artifact-headroom recovery rather than by changing GPTQ calibration. Figure 12 traces the resulting flow. J.3 NanoChat-D12 final recipe The starting recipe is the unmodified upstream NanoChat-D12 pretraining script at the pinned commit, calibrated to a CORE score of 0.1618 in our GPU environment. The inherited stack uses a frozen BPE tokenizer, a 12 layer transformer body that mixes short masked-SDPA sliding-window layers with longer Flash-SDPA layers in an SSSL pattern, a Muon-plus-AdamW optimizer split, a fixed 90 minute training cap, a fixed mix of pretraining-stage and midtraining-stage tokens, and a final language-model head with no learnable bias path on top of lm_head. The final 0.2244 recipe keeps the optimizer and the 90 minute cap and changes three components in code. The first change is a runtime attention-path rewrite. The 12 body layers move from the SSSL pattern to a uniform L pattern, so every layer uses Flash SDPA in the local GPU environment. This removes the runtime tax that the masked-SDPA sliding-window layers paid on this hardware and recovers a measurable amount of wallclock under the same 90 minute cap. The recovered wallclock turns into more training tokens. The recipe expands the data ratio across the pretraining, midtraining, and small final stages to roughly 12 to 100 to 130, which the loop tuned by submitting a sequence of trials that each increased the ratio while watching whether the run still fit inside the cap. The third change is a learnable logit-bias path inserted afterlm_headwith a zero initialization, which acts as a vocabulary-level prior the model can learn during the same fixed run. CORE rises from the runtime jump to 0.1695 at the first keep, then to 0.2029 once the recovered wallclock is spent on more tokens, then to 0.2241 and 0.2244 as the data ratio and the logit-bias path are added on top. Figure 13 traces the resulting flow. J.4 CIFAR-10 Airbench96 final recipe The starting recipe is the unmodified Airbench96 release, calibrated under our ten-seed cold-process protocol to a 26.3560 second mean wallclock at a strict 0.96 mean-accuracy gate. The inherited stack is a fast CIFAR-10 ConvNet, an SGD-style training loop with a linear warmup and cosine decay, a 10 percent warmup ratio, an in-script logging path that runs a small validation evaluation after every epoch, a fixed 45 epoch horizon, and a calibrated learning rate. The final 25.1464 second recipe keeps the network and the optimizer family and changes four components in code. 25 Figure 13: NanoChat-D12 final recipe schematic. The figure summarizes the attention-path rewrite, data-stage ratio expansion, zero-initialized logit-bias path, and CORE trajectory. Post hoc Claude Design-generated explanatory schematic; not part of the search loop. The first change skips most of the in-script logging-only validation calls and keeps only an end-of- training check plus an occasional intermediate one every several epochs, since the harness already runs the strict gate from outside and a per-epoch internal estimate is not needed. The second change shortens the training horizon to 42 epochs, which is the shortest horizon the loop found that still cleared the 0.96 gate after the other speed-recipe changes were composed. The third change raises the peak learning rate to 11, compensating for the shorter horizon by spending more update magnitude per epoch. The fourth change repairs the accuracy margin by reducing the warmup ratio from 10 percent to 5 percent, so the schedule reaches its peak earlier and the body of training has a longer high-rate window. The repair was triggered by exp_060, which had already reached 25.1650 seconds with the peak learning rate raised, but missed the gate at 0.95956 mean accuracy. exp_070 kept the same speed recipe and changed only the warmup ratio, reaching 25.1464 seconds at 0.96008 mean accuracy. Figure 14 traces the resulting flow. K Broader impacts and asset licenses Broader impacts. The positive impact of this work is a more auditable path for empirical ML research. A closed feedback loop records hypotheses, code edits, evaluator outcomes, and failures, so follow-up work can inspect how a result was developed rather than only seeing a final recipe. It may also reduce the cost of improving small training recipes by spending bounded compute on externally verified experiments. The negative impact is that the same automation pattern could accelerate benchmark overfitting, waste compute if attached to poorly designed objectives, or optimize a harmful task more quickly when the evaluator rewards the wrong behavior. Our experiments mitigate these risks by using bounded public-style research environments, evaluator-owned scoring, no private data collection, no deployment-facing model release, and trace archives that expose both successful 26 Figure 14: CIFAR-10 Airbench96 final recipe schematic. The figure summarizes the schedule rewrite, four code-level edits, strict gate enforcement, and closed-loop warmup repair trajectory. Post hoc Claude Design-generated explanatory schematic; not part of the search loop. and failed attempts. Applying the loop to sensitive domains would require stronger human review, access control, and objective auditing than the benchmark settings used here. Existing assets. Table 10 lists the external assets used by the experiments. We do not redistribute raw FineWeb/CommonCrawl text, CIFAR-10 images, upstream NanoChat evaluation datasets, Claude model weights, or third-party benchmark data. The public repository athttps://github.com/cxc scmu/Auto-Research-Recipescontains the harness code, prompt templates, trace metadata, final recipes, release documentation, and pointers for users to obtain third-party assets under their own terms. L Releasable trace contents A frozen run produces ablackboard/directory. The public repository athttps://github.com/c xcscmu/Auto-Research-Recipes releases the subset needed to inspect the reported trajectories and final recipes without exposing raw runtime telemetry: • results.tsv— one row per submitted trial with proposing role, hypothesis, parent exp_id, status, measured score,∆vs prior best, train / eval / total wallclock seconds, packed artifact bytes, and harness notes. Append-only. 27 Table 10: External assets used in the experiments and the license or terms we rely on. AssetUse in this paperLicense or terms OpenAIParameter Golf [OpenAI, 2025] Challenge harness, starting recipe, fixed task protocol, and public starting score. MIT License for the public repos- itory.Challenge data are used through the task harness and are not redistributed. FineWeb/Common- Crawl [Penedo et al., 2024] Fixed language-model data source under- lying the Parameter Golf task and related recipe context. Open Data Commons Attribution License (ODC-By) v1.0, subject to CommonCrawl Terms of Use. nanochat[Karpathy, 2025] NanoChat-D12 starting recipe, ven- dored editable Python tree, and training pipeline. MIT License. Upstream evaluation datasets used by nanochat retain their original terms and are not re- distributed. DataComp-LM / DCLM CORE [Li et al., 2024] CORE-styleevaluationtargetfor NanoChat-D12. MIT License for the framework code. Evaluation subdatasets retain their original licenses and terms. cifar10-airbench [Jordan, 2024] CIFAR-10 Airbench96 starting recipe and speed-run structure. MIT License. CIFAR-10 [Krizhevsky, 2009] Image-classification data for the Air- bench96 environment. C BY 4.0 in UCI dataset metadata (https://doi.org/10.24432/C 5889J). Raw images are not redis- tributed by this paper. Anthropic Claude API models Language-agent proposal generation and post hoc Claude Design explanatory schematics. Anthropic Commercial API service terms (https://w.anthropi c.com/legal/commercial-t erms). Model weights are not ac- cessed or redistributed. • tree.tsv— the same rows in preorder-sorted form withdepthand slash-joinedpath columns so subtrees are contiguous and a single awk can slice an entire branch. • best.json— the current-best row at the moment of write. Updated atomically every time a new keep lands. • KNOWLEDGE.mdandLEADERBOARD.md— de-identified lineage summaries and top keep rows used for audit and compact replay of the reported trajectories. • snapshots/<exp_id>_<role>/ — frozen keep-time or final-recipe workdir snapshots for the reported developed recipes and controls. • Prompt templates, harness code, and reproduction notes in the public repository. The public repository intentionally omits full per-trial stdout, raw runtime event logs, full per-session rendered prompts, full submitted-trial code snapshots, scratch workdirs, and supervisor telemetry. Those files are not needed to inspect the reported score trajectory or final recipes and may contain low-level runtime accounting. The released archive is sufficient to audit the submitted trial rows, follow parent-child lineage, compare released keep/final code snapshots with the reported recipes, and reproduce the final submitted solutions after preparing the third-party task assets under their original terms. 28