Paper deep dive

Discovering Cooperative Pipelines: Autoresearch for Sequential Social Dilemmas

Víctor Gallego

Year: 2026Venue: arXiv preprintArea: cs.MAType: PreprintEmbeddings: 89

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 7/8/2026, 10:28:44 PM

Summary

The paper introduces a two-level autoresearch framework where an outer-loop AI researcher agent autonomously designs and optimizes the inner-loop pipeline of an LLM-based policy synthesizer for multi-agent Sequential Social Dilemmas (SSDs). Evaluated across the Cleanup and Gathering games with utilitarian efficiency and Rawlsian maximin objectives, the framework consistently outperforms hand-designed baselines and prompt-only optimization methods. The discovered pipelines are objective-dependent: optimizing for maximin autonomously injects explicit fairness mechanisms (e.g., duty rotation), supporting an information-design interpretation where the researcher tailors information flow to guide boundedly rational synthesizers.

Entities (9)

Sequential Social Dilemmas (SSDs) · game-environment · 97%Researcher Agent R · ai-agent · 95%Policy Synthesizer LLM M · llm-model · 94%Cleanup · game-environment · 92%Gathering · game-environment · 92%Rawlsian Maximin · welfare-objective · 90%Utilitarian Efficiency · welfare-objective · 90%Information Design · theoretical-concept · 88%GEPA · optimization-method · 85%

Relation Signals (8)

Sequential Social Dilemmas (SSDs) → includes → Cleanup

confidence 95% · We study two canonical SSDs... Cleanup... is a public goods provision game

Sequential Social Dilemmas (SSDs) → includes → Gathering

confidence 95% · Gathering... is a common pool resource game

Researcher Agent R → modifies → Inner-loop Pipeline

confidence 95% · an outer-loop AI agent autonomously redesigns the inner-loop pipeline of an LLM policy-synthesis system

Pipeline Configuration → consistsof → System Prompt

confidence 93% · c=(p,ϕ,ℋ,ι) where p is the system prompt, ϕ is the feedback construction function, ℋ is the helper function library, and ι specifies the iteration logic

Autoresearch Framework → outperforms → Hand-designed Baseline

confidence 92% · the researcher reliably exceeds hand-designed baselines, sharply tightens run-to-run variance

Autoresearch Framework → outperforms → GEPA

confidence 88% · outperforms prompt-only optimization... GEPA optimizes only the system prompt, whereas our researcher modifies the full pipeline

Rawlsian Maximin → triggers → Explicit Fairness Mechanisms

confidence 87% · only under maximin does the researcher inject an explicit fairness mechanism into synthesizer pipelines

Researcher Agent R → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We study two-level autoresearch for cooperation: an outer-loop AI agent autonomously redesigns the inner-loop pipeline of an LLM policy-synthesis system for multi-agent Sequential Social Dilemmas (SSDs). A researcher agent $\mathcal{R}$ (run as a coding agent) reads the inner-loop source code, edits system prompts, feedback functions, helper libraries, and iteration logic, runs evaluations, and decides what to keep, following the autoresearch paradigm. Across two games (Cleanup and Gathering), two policy-synthesizer LLMs, and two welfare objectives (utilitarian efficiency and Rawlsian maximin), the researcher reliably exceeds hand-designed baselines, sharply tightens run-to-run variance, and outperforms prompt-only optimization. The discovered pipelines are objective-dependent: only under maximin does the researcher inject an explicit fairness mechanism into synthesizer pipelines, a class of mechanism that is absent from its own objective-agnostic system prompt and from every efficiency-optimized pipeline. This supports an information-design reading in which the researcher chooses what to reveal to the boundedly rational synthesizer as a function of the welfare objective. Code at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

89,123 characters extracted from source content.

Expand or collapse full text

Discovering Cooperative Pipelines: Autoresearch for Sequential Social Dilemmas Víctor Gallego Komorebi AI Technologies victor.gallego@komorebi.ai Abstract We study two-level autoresearch for cooperation: an outer-loop AI agent autonomously redesigns the inner-loop pipeline of an LLM policy-synthesis system for multi-agent Sequential Social Dilemmas (SSDs). A researcher agent ℛR (run as a coding agent) reads the inner-loop source code, edits system prompts, feedback functions, helper libraries, and iteration logic, runs evaluations, and decides what to keep, following the autoresearch paradigm. Across two games (Cleanup and Gathering), two policy-synthesizer LLMs, and two welfare objectives (utilitarian efficiency and Rawlsian maximin), the researcher reliably exceeds hand-designed baselines, sharply tightens run-to-run variance, and outperforms prompt-only optimization. The discovered pipelines are objective-dependent: only under maximin does the researcher inject an explicit fairness mechanism into synthesizer pipelines, a class of mechanism that is absent from its own objective-agnostic system prompt and from every efficiency-optimized pipeline. This supports an information-design reading in which the researcher chooses what to reveal to the boundedly rational synthesizer as a function of the welfare objective. Code at https://github.com/vicgalle/autoresearch-social-dilemmas. 1 Introduction Sequential Social Dilemmas (SSDs) Leibo et al. (2017) are the multi-agent analogue of the prisoner’s dilemma in temporally rich Markov games: individually rational play leads to collectively suboptimal outcomes through pollution, over-harvesting, or open conflict. Standard multi-agent reinforcement learning (MARL) struggles in this regime due to credit assignment, non-stationarity, and large joint action spaces Buşoniu et al. (2008). A complementary approach, recently introduced by Gallego Gallego (2026), sidesteps these difficulties by replacing decentralized parameter-space optimization with centralized algorithm-space synthesis: a frozen LLM writes a Python policy function, evaluates it in self-play, and iteratively refines it from performance feedback. A single generation step can produce coordination logic (territory partitioning, role assignment, conditional cooperation) at a sample efficiency several orders of magnitude beyond what gradient-based MARL achieves on the same environments. This shifts where the design problem lives, rather than removing it. The inner-loop pipeline that drives the synthesizer has many free parameters: which system prompt, which feedback variables, which helper functions, how many refinement steps. Each materially affects the resulting policies, and prior work tuned them by hand. A natural question follows: can an AI agent design the pipeline? We answer affirmatively with a two-level autoresearch framework. An outer-loop researcher agent (Claude Opus 4.6, run as a coding agent) edits the source files of an inner-loop policy synthesizer (another LLM), runs evaluations on held-out seeds, and keeps modifications that improve a fixed welfare objective Φ . The outer agent operates on an ordinary git repository (reading code, writing diffs, running shell commands, etc) without task-specific scaffolding beyond a standard CLI and git, mirroring the autoresearch paradigm of Karpathy Karpathy (2026) for single-GPU LLM pretraining. Although the inner-loop SSDs are gridworld benchmarks rather than physical systems, the outer-loop discovery process itself runs under conditions a deployed discovery agent faces: noisy multi-seed evaluations, stochastic code generation, an LLM-evaluation budget that bounds how often Φ can be queried, and a heterogeneous code repository the agent must navigate end-to-end on its own. Our contributions are: i) a general two-level framework that delegates the design of an LLM synthesis pipeline to a coding agent operating on a real software repository (Section 3); i) the first instantiation of the autoresearch paradigm in a multi-agent decision-making domain, with experiments across two SSDs (Cleanup and Gathering), two policy LLMs, and two welfare objectives (utilitarian efficiency U and Rawlsian maximin mini⁡Ri _iR_i) (Section 4); and i) a mechanism-design interpretation supported by the qualitatively different pipelines the agent produces under different welfare objectives, including the autonomous insertion of explicit fairness mechanisms (usually time-based duty rotation) into the researcher-authored synthesizer prompts and helpers, in every maximin run and no efficiency run. 2 Background We build on the iterative LLM policy synthesis framework of Gallego Gallego (2026), which serves as the (frozen) inner loop of our two-level system. This section recalls the SSD formalism, the social outcome metrics, and the base synthesis loop. 2.1 Sequential Social Dilemmas A Sequential Social Dilemma is a partially observable Markov game =⟨N,,ii=1N,T,Rii=1N,H⟩G= N,S,\A_i\_i=1^N,T,\R_i\_i=1^N,H with N agents, state space S (the gridworld configuration), per-agent action spaces iA_i, transition function T, reward functions RiR_i, and episode horizon H Leibo et al. (2017). Beyond the dilemma’s matrix-game structure, SSDs add temporal richness: agents must learn when and where to cooperate, not just whether to. We study two canonical SSDs that capture complementary dilemma types. Cleanup Hughes et al. (2018) is a public goods provision game (N=10N=10). The map has two regions: a river that accumulates waste, and an orchard where apples grow. Apples regrow only when river pollution is below threshold. Each agent can fire a cleaning beam (cost −1-1) that removes waste, collect apples (+1+1 each), or fire a tagging beam (cost −1-1, inflicting −50-50 on the target and removing it for 2525 steps). The dilemma: cleaning is costly but its benefits are public, so purely self-interested agents free-ride. Gathering Leibo et al. (2017); Perolat et al. (2017) is a common pool resource game (N=4N=4). Agents collect shared apples on a fixed respawn timer and may fire tagging beams to temporarily remove rivals. The dilemma: agents can coexist and share resources, or aggress to monopolize them; aggression wastes time and reduces total welfare. Both games use 88–99 discrete actions (movement, rotation, beam, stand, optionally clean) and episodes of H=1000H=1000 steps. The two dilemmas differ in cost structure: asymmetric provision (cleaners pay, all benefit) vs. symmetric restraint (every agent faces the same temptation), a distinction that drives our experimental findings. Social metrics. Following Perolat et al. Perolat et al. (2017), let Ri=∑t=0H−1ritR_i= _t=0^H-1r_i^t denote agent i’s episode return. We evaluate four social outcomes: U U =1H∑i=1NRi = 1H _i=1^NR_i (efficiency) E E =1−∑i,j|Ri−Rj|2N∑iRi =1- _i,j|R_i-R_j|2N _iR_i (equality) S S =1N∑i=1Nt¯i = 1N _i=1^N t_i (sustainability) P P =1H∑t=0H−1|i:activeit| = 1H _t=0^H-1 |\i: active_i^t\ | (peace) where t¯i t_i is the mean timestep at which agent i collects positive reward (higher ⇒ resources preserved later) and activeit active_i^t indicates that agent i is not tagged out at step t. We additionally consider the maximin (Rawlsian) welfare criterion mini⁡Ri _iR_i, which optimizes the worst-off agent’s return and serves as the second objective in our experiments. A note on the dilemma’s status under symmetric programmatic policies. We adopt the SSD environments of Leibo et al. (2017); Hughes et al. (2018); Perolat et al. (2017) as benchmarks, but in our synthesis setup a single Python function π controls all N agents (§3). This reframes the strategic problem: the individual-rationality constraint that makes classical SSDs a dilemma is replaced by a joint coordination/scheduling problem with the welfare objective Φ as the explicit target. Locally myopic per-agent code can still recreate dilemma-shaped behavior (and the baseline pipeline in fact does), but cooperation here is a joint-optimization outcome, not an equilibrium under individual rationality. We interpret the mechanisms the researcher discovers (duty rotation, role assignment) accordingly: they are coordination solutions in algorithm space that resemble the fairness mechanisms one would want a decentralized MARL system to converge to, not equilibria induced by self-interested agents. 2.2 Iterative LLM Policy Synthesis Let Π denote the space of code-based policies: deterministic functions π:×[N]→π:S×[N] expressed as executable Python code. Each policy has access to the full environment state and a library of helpers (BFS pathfinding, beam targeting, coordinate transforms). This state access is a deliberate design choice: programmatic policies operate in algorithm space rather than the reactive observation-to-action space of neural policies, which lets a single LLM generation step encode rich coordination logic. A frozen LLM ℳM acts as the policy synthesizer. Given a system prompt p describing the environment API and a feedback prompt qkq_k, it produces a new policy πk+1=ℳ(p,q(πk,ℱk)), _k+1=M\! (p,\;q( _k,F_k) ), where πk _k is the previous policy (its source code) and ℱkF_k is the evaluation feedback. All N agents execute the same program πk _k in self-play. We stress that this is symmetric, not behaviorally homogeneous: since πk _k takes agent_id as an argument, a single shared program can induce distinct per-agent behaviors (cleaner vs. gatherer assignment, time-rotated duty cycles, partitioned territories; see the synthesized policies in Appendix D). What is shared is the source code; the sampled action distributions can differ across agents. Evaluation over a set of random seeds S yields the mean per-agent return r¯k r_k and the social metrics vector k=(Uk,Ek,Sk,Pk)m_k=(U_k,E_k,S_k,P_k). Each generated policy passes an AST-based safety check (blocking eval, file I/O, network access) followed by a short smoke test; failures trigger regeneration (up to R attempts) with the error message appended to the prompt. Feedback. We package the previous policy’s code together with all available evaluation signals: ℱk=(code(πk),r¯k,k,),F_k= ( code( _k),\; r_k,\;m_k,\;d ), (1) where d contains natural-language definitions of each social metric. The LLM consumes ℱkF_k to revise and improve the policy. This is a starting point; the choice of feedback content is a single design decision within a much larger pipeline configuration space: our two-level framework (Section 3) opens the full space to automated search. 3 Two-Level Framework Inner Loop (K iterations)Researcher ℛR (Claude Opus 4.6)Pipeline cjc_j p,ϕ,ℋ,ιp,\;φ,\;H,\; ℳMEvaluate in GFeedback ℱkF_kMetrics: JjJ_j, jm_j, Δj _jJj>J∗+τJ_j>J^*+τ ? keep ∣ discardmodifyhistoryj=1,…,Jj=1,…,J Figure 1: Two-level automated research framework (Algorithm 1). Algorithm 1 Two-Level Automated Research 1:Game G, policy synthesizer ℳM, researcher ℛR, system prompt pℛp_R, initial config c0c_0, outer iterations JmaxJ_ , welfare objective Φ , held-out seeds ShoS_ ho, keep threshold τ≥0τ≥ 0 2:Best configuration c∗c^*, best policy π∗π^* 3:π0∗←InnerLoop(ℳ,,c0)π^*_0← InnerLoop(M,G,c_0) 4:J0←Φ(Eval(π0∗;,Sho))J_0← ( Eval(π^*_0;\,G,\,S_ ho)) 5:c∗←c0,J∗←J0c^*← c_0,\;J^*← J_0 // running best 6:history←(c0,J0,0,∅)history←\(c_0,J_0,m_0, )\ 7:for j=1,…,Jmaxj=1,…,J_ do 8: cj←ℛ(pℛ,code(c∗),history)c_j \! (p_R,\, code(c^*),\,history ) 9: if ¬ValidateConfig(cj) \, ValidateConfig(c_j) then retry ≤R≤ R 10: πj∗←InnerLoop(ℳ,,cj)π^*_j← InnerLoop(M,G,c_j) 11: Jj←Φ(Eval(πj∗;,Sho))J_j← ( Eval(π^*_j;\,G,\,S_ ho)) 12: Δj←Diff(c∗,cj) _j← Diff(c^*,c_j) // code diff 13: if Jj>J∗+τJ_j>J^*+τ then keep: c∗←cj,J∗←Jjc^*← c_j,\;J^*← J_j; else discard // τ=0τ=0 in our runs 14: history←history∪(cj,Jj,j,Δj)history ∪\(c_j,J_j,m_j, _j)\ 15:end for 16:return c∗,πc∗c^*,\,π^*_c^* We introduce a two-level system where a researcher agent ℛR autonomously discovers configurations that optimize the output of an inner-loop system. While we instantiate this for multi-agent policy synthesis, the architecture is general: any pipeline where an LLM generates artifacts, evaluates them, and iterates can serve as the inner loop. The fundamental insight is that the entire inner-loop codebase is a designable artifact that a code-based agent can search over. Figure 1 illustrates the architecture. 3.1 Configuration Space Let C denote the space of pipeline configurations. Each configuration c∈c specifies the full inner-loop setup: c=(p,ϕ,ℋ,ι)c=(p,\,φ,\,H,\, ) (2) where p is the system prompt, ϕφ is the feedback construction function (which metrics and diagnostics to include, how to frame it, whether to inject adaptive hints and thresholds, etc), ℋH is the helper function library (auxiliary functions for pathfinding, getting aggregates of useful environemnt quantities, etc.), and ι specifies the iteration logic (number of inner iterations K, sampling strategy). Table 1 provides concrete examples. The validation pipeline is part of the frozen inner-loop infrastructure rather than a configurable component, the researcher cannot modify it in our experiments to prevent reward hacking. Table 1: Configuration components modifiable by the researcher, with concrete edits that ℛR made in our runs. The environment simulator, ground-truth evaluation, and policy LLM weights are frozen. Scope Concrete edits by ℛR (see appendix) p System prompt Multi-section strategic briefing with explicit duty-rotation template (agent_id + step//T) % n (App. C.3, Listing 4) ϕφ Feedback fn Thresholded diagnostics: “FAIRNESS ALERT” fires when mini⁡Ri<0 _iR_i<0; “DO NOT REGRESS” guard fires when U≥2.5U≥2.5 (App. C.4, Listing 5) ℋH Helper library BFS-Voronoi territories with respawn-timer awareness; band-based apple zoning by agent_id (App. C.2, Listings 3, 2) ι Iteration logic Per-condition K∈2,3K∈\2,3\; |S||S| walked 5→8→125→8→12 for variance control; thinking budget 1010–3232k (App. C.5, Table 7) The hand-designed feedback of Gallego (2026) corresponds to a single fixed instantiation of ϕφ. Our framework opens the full configuration space to automated search. 3.2 Inner Loop (Policy Synthesis) Given a configuration c, the inner loop executes K iterations of LLM policy synthesis: πc∗=InnerLoop(ℳ,,c)π^*_c= InnerLoop(M,G,c) (3) Each iteration k proceeds in four stages, following Gallego (2026): 1. Synthesize. The policy LLM ℳM receives the system prompt p, the previous policy’s source code πk−1 _k-1, and feedback ℱk−1F_k-1 constructed by ϕφ. It generates a new Python policy function πk _k that has access to full environment state and the helper library ℋH. 2. Validate. The generated code undergoes AST-based safety checks (blocking dangerous operations such as file I/O and network access) followed by a short smoke test. Failures trigger re-generation (up to R retries), with the error message appended to the prompt. 3. Evaluate. All N agents execute the same policy πk _k in self-play over |S||S| random seeds (note the policy is conditional on agent_id). The evaluation yields the mean per-agent reward r¯k r_k and the social metrics vector k=(Uk,Ek,Sk,Pk)m_k=(U_k,E_k,S_k,P_k). 4. Feedback. The feedback function ϕφ constructs the prompt for the next iteration from (πk,r¯k,k)( _k, r_k,m_k), packaging the previous policy’s code together with the scalar reward, the social metrics vector, their natural-language definitions, and any adaptive diagnostics that ϕφ injects. The inner loop output is scored on held-out seeds via the configuration-level map J(c)=Φ(Eval(πc∗;,Sheld-out)),J(c)= \! ( Eval(π^*_c;\;G,\;S_held-out) ), (4) where Eval returns the per-agent returns of πc∗π^*_c in G averaged over the held-out seeds, and Φ is a fixed welfare functional that aggregates those returns into a scalar. We consider two alternative welfare functionals: ΦU _U =U=1H∑i=1NRi =U= 1H _i=1^NR_i (utilitarian efficiency) (5) Φmin _ =mini⁡Ri = _iR_i (Rawlsian maximin) (6) ΦU _U rewards collective throughput and is indifferent to how reward is distributed across agents, whereas Φmin _ instead optimizes for the worst-off agent, pressuring the researcher toward configurations that distribute the cost of cooperation. The researcher’s goal is to maximize J(c)J(c) for a chosen Φ ; we use Φ to denote the welfare objective throughout, and J for the per-configuration scalar score returned by held-out evaluation. 3.3 Outer Loop (Automated Research) The researcher agent ℛR iteratively modifies the pipeline configuration. Following the autoresearch paradigm Karpathy (2026), ℛR operates on the inner-loop codebase as a modifiable artifact, proposing changes, observing outcomes, and refining. The procedure is formalized in Algorithm 1. At each outer iteration j, the researcher ℛR receives: i) the full source code of the current running-best configuration c∗c^* (prompts, feedback construction, helpers, iteration logic); i) the experiment history: for each prior iteration, the code diff Δi _i, ground-truth score JiJ_i, social metrics vector im_i, and whether the iteration was kept or discarded; i) the environment source code (read-only), enabling the researcher to reason about game mechanics. Discarded iterations are reverted on disk (git checkout -- pipeline/) so that the next proposal cj+1c_j+1 is constructed on top of c∗c^*, not on top of cjc_j. The researcher proposes a new configuration cjc_j by generating code modifications. Concretely, ℛR is a coding agent (Claude Code CLI) that operates on a real software repository: it reads and edits Python source files, runs shell commands, inspects evaluation outputs, and commits changes to a dedicated git branch, following the same workflow a human researcher would follow. 3.4 Connection to Automated Mechanism Design The two-level structure admits a mechanism design interpretation. The researcher ℛR acts as a mechanism designer: it controls the information structure (what metrics to reveal, how to frame them), the action space (what helper functions are available), and the incentive structure (how feedback is presented) under which the policy synthesizer ℳM operates. The synthesizer acts as the agent within the designed mechanism. This connects to the automated mechanism design literature Conitzer and Sandholm (2002), where a principal designs rules to induce desired behavior from self-interested agents. In our setting: (i) the principal is the researcher ℛR, optimizing the ground-truth score J induced by the welfare objective Φ ; (i) the agent is the synthesizer ℳM, optimizing per-agent reward as instructed; (i) the mechanism is the configuration c: prompts, feedback, helpers, iteration logic; (iv) the outcome is the social welfare of the resulting multi-agent policy πc∗π^*_c. A crucial distinction from classical mechanism design: ℳM follows instructions but has bounded rationality in the sense that its ability to synthesize effective policies depends on the information and tools provided. The researcher’s task is thus closer to information design Kamenica and Gentzkow (2011): choosing what to reveal to help ℳM navigate the cooperation–defection tension. Our experiments (Section 4) show that the researcher designs qualitatively different information structures depending on the welfare objective Φ , supporting this interpretation empirically. 4 Experiments We conduct 12 autonomous researcher runs across a factorial design. The researcher agent ℛR is Claude Opus 4.6, invoked via the Claude Code CLI as a coding agent. Each run operates on a dedicated git branch of a real Python codebase: ℛR edits source files in pipeline/, executes evaluation scripts, reads metric outputs, and iterates, without human intervention. Design. For Cleanup: 22 policy LLMs × 22 objectives × 22 replications == 8 runs. For Gathering: 22 policy LLMs × 11 objective × 22 replications == 4 runs. Maximin runs are unnecessary for Gathering because efficiency optimization alone achieves close to perfect equality. Models. Policy synthesizer ℳM: Gemini 3.1 Pro (Google) or Claude Sonnet 4.6 (Anthropic), to recent state-of-the-art LLMs. Both use extended thinking. We additionally test with Gemma 4 26B-A4B-IT (Google), a smaller open-weight model, to probe the framework’s behavior when the policy synthesizer has substantially lower capability (Appendix B.3). Baselines. The hand-designed feedback configuration from prior work Gallego (2026) serves as the initial pipeline c0c_0 for all runs. On Cleanup (N=10N=10), this baseline achieves U=1.93U=1.93/2.702.70 (mean/max) with Gemini and U=0.86U=0.86/1.561.56 with Sonnet. We additionally compare against GEPA Agrawal and others (2026), an automated prompt optimization method that iteratively refines the system prompt via LLM reflection. GEPA optimizes only the system prompt p, whereas our researcher modifies the full pipeline (p,ϕ,ℋ,ι)(p,φ,H, ). We give GEPA a matched compute budget: the same number of optimization steps as outer iterations used by our method, so all in all both methods use a comparable number of environment evaluations. 4.1 Results Table 2 presents the main results, comparing our autoresearch framework against the hand-designed baseline of Gallego (2026) and GEPA Agrawal and others (2026). Table 2: Results. Mean / max across the runs per condition. Bold marks the best mean per metric within each Policy LLM × Game sub-block. Baseline: unmodified, hand-designed pipeline from Gallego (2026). GEPA: automated prompt optimization Agrawal and others (2026) with matched compute budget. Policy LLM Target U E mini⁡Ri _iR_i Cleanup — Baseline Gemini 3.1 Pro — 1.93/2.70 0.17/0.62 −-159/−-84 Sonnet 4.6 — 0.86/1.56 −-0.02/0.25 −-151/−-59 Cleanup — GEPA Agrawal and others (2026) Gemini 3.1 Pro ΦU _U 1.34/1.37 −-0.10/−-0.05 −-149/−-126 Φmin _ 1.76/2.76 0.90/0.96 143/245 Sonnet 4.6 ΦU _U 1.04/1.20 −-0.59/−-0.21 −-164/−-126 Φmin _ −-0.63/1.60 0.77/1.00 −-147/6 Cleanup — Two-level Autoresearch Gemini 3.1 Pro ΦU _U 3.20/3.25 0.55/0.61 −-211/−-182 Φmin _ 3.16/3.19 0.98/0.98 290/296 Sonnet 4.6 ΦU _U 3.12/3.14 0.66/0.70 −-196/−-146 Φmin _ 2.57/2.93 0.91/0.97 179/200 Gathering — Baseline Gemini 3.1 Pro — 2.04/2.42 0.90/0.98 412/571 Sonnet 4.6 — 0.03/0.03 0.54/0.54 0/0 Gathering — GEPA Agrawal and others (2026) Gemini 3.1 Pro ΦU _U 2.08/2.35 0.94/0.96 436/518 Sonnet 4.6 ΦU _U 1.20/1.23 0.63/0.71 86/123 Gathering — Two-level Autoresearch Gemini 3.1 Pro ΦU _U 2.49/2.51 0.98/0.98 582/582 Sonnet 4.6 ΦU _U 2.52/2.52 0.96/0.98 576/597 Finding 1: The researcher reliably improves over hand-designed baselines and outperforms prompt-only optimization. Every run improves substantially regardless of starting point (Figure 2). On Cleanup, autoresearch lifts both LLMs to U≈3.1U≈ 3.1–3.23.2 from baselines of 1.931.93 (Gemini) and 0.860.86 (Sonnet), nearly closing the gap between them; on Gathering, all four runs converge to U∈[2.47,2.52]U∈[2.47,2.52] from baselines spanning 0.030.03–2.422.42. Run-to-run spread is tight (gaps within 0.050.05 on Cleanup-ΦU _U), suggesting the researcher reliably finds the performance ceiling of each policy LLM via the pipeline modifications it discovers (helpers, prompts, and feedback; see appendices C.2–C.4 for a selection of them). At matched environment queries, autoresearch beats GEPA by 22–3×3× on Cleanup (both ΦU _U and Φmin _ ) and 20%20\% on Gathering, with the gap widening for the weaker policy LLM: GEPA-Sonnet under Φmin _ can collapse to a pathological “everyone cleans, nobody eats” regime (U=−2.87U=-2.87, E=1.00E=1.00), while autoresearch-Sonnet reaches mini⁡Ri≈200 _iR_i≈ 200 reliably. Modifying the full pipeline, not just the prompt, is what closes these gaps. Figure 2: Efficiency (U) across researcher iterations for all 12 runs. Left: Cleanup with 8 runs across 2 LLMs × 2 objectives. Solid lines connect kept iterations (those for which Jj>J∗+τJ_j>J^*+τ strictly exceeded the running-best score; τ=0τ=0 in our runs); open circles mark discarded iterations (Jj≤J∗J_j≤ J^*), which are reverted on disk before the next proposal. Dashed horizontal line: hand-designed baseline from Gallego (2026). All runs converge to U≈3.1U≈ 3.1–3.23.2 despite diverse starting points. Right: Gathering with 4 runs (2 per LLM). All converge to U≈2.5U≈ 2.5 within 2–4 iterations. Finding 2: No efficiency–fairness tradeoff in Cleanup (Gemini). Maximin-optimized Gemini pipelines sacrifice only 1%1\% efficiency (U: 3.163.16 vs. 3.203.20) while achieving near-perfect equality (E: 0.980.98 vs. 0.550.55) and transforming maximin from deeply negative baselines (−99-99 to −84-84) to mini⁡Ri=290 _iR_i=290 (Figure 3). The researcher discovers fair duty rotation (Listing 7; primed by the prompt rewrite of Listing 4)—time-based cycling using agent_id and env._step_count—simultaneously improving worst-off welfare and collective output. Because cleaning is a public good, distributing the cleaning cost fairly ensures enough cleaners to sustain apple production. For Sonnet, there is a moderate tradeoff: efficiency drops from 3.123.12 to 2.572.57 under maximin optimization. The gap reflects Sonnet’s harder time implementing complex coordination mechanisms (role rotation, zone assignment) from strategic hints alone. Finding 3: Game structure determines whether fairness requires explicit optimization. In Cleanup, where cleaning costs are borne asymmetrically (cleaners pay −1-1, free-riders collect apples), baseline equality ranges from E=0.04E=0.04 to 0.620.62, and maximin optimization is required to reach E>0.9E>0.9. In Gathering, where all agents face a symmetric landscape, efficiency optimization alone achieves E>0.94E>0.94 across all 4 runs: no separate maximin runs are needed. This generalizes: in provision dilemmas with asymmetric costs, fairness requires designed mechanisms (role rotation, duty sharing); in restraint dilemmas with symmetric costs, fairness emerges as a free byproduct of efficient coordination. The researcher independently discovers this, it creates role differentiation pipelines only for Cleanup, and pure spatial-coordination pipelines for Gathering. Finding 4: Convergent discovery of qualitatively different strategies per objective. Despite fully independent runs, the researcher converges on the same core strategies within each condition (Table 3, Appendix B). In Cleanup, waste-counting helpers and spatial zone partitioning appear across all runs. The qualitative dividing line is the presence of an explicit fairness mechanism, which appears in 4/4 maximin runs but 0/4 efficiency runs. In 3/4 maximin runs (both Gemini runs and one Sonnet run, Listings 7, 8) the researcher writes time-based role rotation into the synthesizer prompt; in the remaining maximin run the researcher writes a structurally distinct “collective threshold” mechanism in which all agents synchronously switch between cleaning and collecting based on waste_fraction(env) (achieving comparable maximin without an agent-index phase). Under efficiency optimization, the researcher instead writes static role assignment (some agents always clean), producing high collective output at the cost of equality (Listing 6). In Gathering, the researcher discovers BFS-Voronoi territory partitioning and respawn-timer awareness (Listings 3, 9), with no role differentiation: optimization is purely spatial (who collects which apples) and temporal (respawn-aware positioning). The convergence here is at the level of which artifacts ℛR injects into p, ϕφ, ℋH. Once the researcher-authored prompt contains a rotation template (e.g., Listing 4), the downstream synthesizer’s implementation is unsurprising. The non-trivial claim is that ℛR writes such a template only under Φmin _ , never under ΦU _U, despite the researcher system prompt pℛp_R being identical across objectives and containing no rotation formula, no objective-conditional guidance, and no link between any strategy class and either welfare criterion (Listing 1, Appendix C.1). Common failure modes. Three named patterns drive most discarded iterations: (1) over-prescription (too many strategic hints confuse ℳM), (2) iteration regression (K∈4,5K∈\4,5\ over-refines a working policy), and (3) feedback overload (verbose per-agent diagnostics cause over-correction). Counts per mode and the residual “pure J-regression” category are tabulated in Appendix B.1. Spec-gaming on the modifiable surface. The tuple c=(p,ϕ,ℋ,ι)c=(p,φ,H, ) is unconstrained, so ℛR could in principle game the held-out J by exposing simulator internals, hard-coding actions, or overfitting hyperparameters to the eval seed set. We inspected the final pipelines from all 12 runs and found no such patterns: ℋH edits are spatial heuristics and state queries, ϕφ edits are thresholded interventions on the optimized metric itself (not on seed-specific values), and ℛR walks |S||S| upward (5→8→125→8→12) when chasing maximin (reducing seed-noise, not exploiting it). The full inspection, including the closest observed near-miss (over-prescription, which manifests as a regression on J rather than an undeserved gain), is reported in Appendix B.2. 5 Related Work Sequential social dilemmas. SSDs were introduced by Leibo et al. Leibo et al. (2017) (Gathering) and extended to public-goods settings by Hughes et al. Hughes et al. (2018) (Cleanup); Perolat et al. Perolat et al. (2017) formalized the social outcome metrics (U,E,S,P)(U,E,S,P) used here. Subsequent work studies inequity aversion, intrinsic motivation, and reputational mechanisms for promoting cooperation in MARL agents. We complement this line by automating the search for cooperative programs rather than evolving neural policies, and by treating the welfare objective Φ as a designable parameter that the outer agent optimizes for, obtaining qualitatively different cooperative behaviors (static role assignment vs. duty rotation) under different objectives without changing the environment. LLMs for policy and program synthesis. FunSearch Romera-Paredes et al. (2024) evolves programs for combinatorial discovery; Eureka Ma et al. (2024) synthesizes reward functions from environment source code; Voyager Wang et al. (2024) and Code as Policies Liang et al. (2023) generate executable skill code for single-agent embodied control; ReEvo Ye et al. (2024) evolves heuristics with reflective feedback. These works target single-agent settings or non-strategic optimization. Gallego Gallego (2026), on which our inner loop is based, extended LLM program synthesis to the multi-agent SSD setting, where one program must coordinate N self-play copies. Our contribution is one level above: rather than tuning the synthesizer for one task, we let an outer agent rewrite the synthesis pipeline itself. LLM reflection and prompt optimization. Reflexion Shinn et al. (2023), Self-Refine Madaan et al. (2023), OPRO Yang et al. (2024), and GEPA Agrawal and others (2026) demonstrate that structured verbal feedback loops improve LLM outputs; ERL Shi et al. (2026) internalizes such reflection via self-distillation. These methods optimize the prompt or the model’s reasoning trajectory in isolation. Our framework instead modifies the entire surrounding pipeline (system prompt, feedback construction, helper library, iteration logic) making prompt optimization one component of a strictly larger search space. Automated AI research. A small but growing body of work delegates the design of ML pipelines to LLM coding agents. Karpathy’s autoresearch Karpathy (2026) runs a coding agent that modifies train.py for nanoGPT pretraining and is rewarded for validation loss. Our system shares this autoresearch architecture (coding agent + frozen evaluation harness + diff-based history) but targets a different inner loop (multi-agent policy synthesis instead of single-model pretraining) and a different objective (multi-agent social welfare instead of validation bits-per-byte). To our knowledge this is the first instantiation of the autoresearch paradigm in a multi-agent decision-making domain. Automated mechanism and information design. Classical automated mechanism design Conitzer and Sandholm (2002) computes optimal allocation rules for self-interested agents under explicit incentive constraints, while information design Kamenica and Gentzkow (2011) studies how a principal commits to a signaling policy that shapes a receiver’s behavior. Our outer agent solves a related but distinct problem: ℳM is not strategically deceptive (it follows instructions) but is boundedly rational, so the researcher must decide what information, helpers, and structure to expose so that ℳM writes policies achieving the principal’s welfare goal. Section 4 shows that the pipelines ℛR produces are genuinely a function of the welfare objective, supporting this information-design framing empirically. 6 Discussion and Conclusion We have presented a two-level framework where a coding agent autonomously discovers pipeline configurations that improve the output of an inner-loop LLM system. Applied to multi-agent policy synthesis in social dilemmas, the researcher agent reliably exceeds hand-designed baselines across multiple independent runs, and converges on qualitatively similar strategies within each game–objective condition. The framework requires no task-specific scaffolding beyond a standard CLI and git: the agent operates on a standard software repository using the same tools (file editing, shell commands, git) available to a human researcher. The inner-loop validation pipeline, helper-library skeleton, and orchestrator API are themselves deliberately scoped artifacts (see App. A). Mechanism design in action. Our results empirically support the mechanism design interpretation of Section 3.4. The researcher acts as an information designer: under ΦU _U, it reveals efficiency-oriented information (waste counts, zone assignments) that guides ℳM toward productive but unequal role allocation (Listing 6). Under Φmin _ , it additionally reveals fairness-oriented structure such as rotation schedules and equity feedback (Listings 4, 5), guiding ℳM toward egalitarian coordination. Human oversight. Our system is fully autonomous by design, but its architecture offers natural affordances for human-in-the-loop oversight: ℛR operates on a standard git repository, fully-auditable. More broadly, the separation between the researcher ℛR and the evaluation Φ creates a delegation boundary: the human defines what to optimize (the welfare objective), while the agent decides how. Future work. First, we are interested in applying the two-level framework to other LLM-driven pipelines (code optimization, scientific experiment design, infrastructure tuning), where the same architecture applies with a different inner loop and objective Φ ; next, adversarial objectives can be intriguing, to test whether the researcher discovers exploitative pipeline configurations, exposing reward hacking risks; and asymmetric programs: extend to settings where agents run different source code (the present setup is symmetric in code but already supports heterogeneous behavior via agent_id). References [1] L. A. Agrawal et al. (2026) GEPA: reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457. Cited by: §4, §4.1, Table 2, Table 2, Table 2, §5. [2] L. Buşoniu, R. Babuška, and B. De Schutter (2008) A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38 (2), p. 156–172. Cited by: §1. [3] V. Conitzer and T. Sandholm (2002) Complexity of mechanism design. In Proc. 18th Conference on Uncertainty in Artificial Intelligence, p. 103–110. Cited by: §3.4, §5. [4] V. Gallego (2026) Cooperation and exploitation in llm policy synthesis for sequential social dilemmas. arXiv preprint arXiv:2603.19453. Cited by: Table 5, §C.4, Table 7, §1, §2, §3.1, §3.2, Figure 2, §4, §4.1, Table 2, §5. [5] E. Hughes, J. Z. Leibo, M. Phillips, K. Tuyls, E. A. Dueñez-Guzmán, A. García Castañeda, I. Dunning, T. Zhu, K. R. McKee, R. Koster, H. Roff, and T. Graepel (2018) Inequity aversion improves cooperation in intertemporal social dilemmas. In Neural Information Processing Systems, Cited by: §2.1, §2.1, §5. [6] E. Kamenica and M. Gentzkow (2011) Bayesian persuasion. American Economic Review 101 (6), p. 2590–2615. Cited by: §3.4, §5. [7] A. Karpathy (2026) Autoresearch: AI agents running research on single-GPU nanochat training automatically. Note: https://github.com/karpathy/autoresearch Cited by: §1, §3.3, §5. [8] J. Z. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel (2017) Multi-agent reinforcement learning in sequential social dilemmas. In Proc. 16th Conference on Autonomous Agents and MultiAgent Systems, p. 464–473. Cited by: §1, §2.1, §2.1, §2.1, §5. [9] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng (2023) Code as policies: language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), p. 9493–9500. Cited by: §5. [10] Y. J. Ma, W. Liang, G. Wang, D. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar (2024) Eureka: human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representations, Cited by: §5. [11] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. (2023) Self-refine: iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems, Cited by: §5. [12] J. Perolat, J. Z. Leibo, V. Zambaldi, C. Beattie, K. Tuyls, and T. Graepel (2017) A multi-agent reinforcement learning model of common-pool resource appropriation. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §2.1, §2.1, §2.1, §5. [13] B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, et al. (2024) Mathematical discoveries from program search with large language models. Nature 625 (7995), p. 468–475. Cited by: §5. [14] T. Shi, S. Chen, B. Jiang, L. Song, L. Yang, and J. Zhao (2026) Experiential reinforcement learning. arXiv preprint arXiv:2602.13949. Cited by: §5. [15] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, Cited by: §5. [16] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2024) Voyager: an open-ended embodied agent with large language models. Transactions on Machine Learning Research. Cited by: §5. [17] C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen (2024) Large language models as optimizers. In The Twelfth International Conference on Learning Representations, Cited by: §5. [18] H. Ye, J. Wang, Z. Cao, F. Berto, C. Hua, H. Kim, J. Park, and G. Song (2024) ReEvo: large language models as hyper-heuristics with reflective evolution. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §5. Appendix A Limitations We flag three limitations of the present study, in roughly decreasing order of how much they bound the claims. 1. Single researcher LLM. All main-experiment runs (and the Gemma appendix) use the same researcher ℛR (Claude Opus 4.6 via the Claude Code CLI). A researcher ablation across ℛR-LLMs, in which the same fixed system prompt pℛp_R (Appendix C.1) is run with several frontier coding agents, is the single most important follow-up. 2. Inner-loop infrastructure is itself a designed artifact. The line “ℛR uses no task-specific scaffolding” refers to the researcher’s tooling (CLI, git, file edits) and to the fact that the configuration c=(p,ϕ,ℋ,ι)c=(p,φ,H, ) starts from a deliberately weak baseline. It does not mean the surrounding system is unscoped: the inner-loop validation pipeline (AST safety checks, sandboxed execution, multi-seed evaluation harness), the helper-library skeleton (pipeline/helpers.py starts non-empty), and the orchestrator API (run_inner_loop.py) are themselves engineered artifacts that bound what ℛR can and cannot do. Generalization of the framework to settings without an equivalent harness is plausible but unevaluated. 3. Gridworld scope. The inner-loop SSDs are 2D gridworlds with fully-observed integer states, discrete action spaces, and short (H=1000H=1000-step) episodes. The specific mechanisms the researcher discovers (BFS-Voronoi partitioning, time-rotated agent-id phase counters, waste-fraction threshold rules) exploit this structure and are gridworld-shaped; their transfer to higher-dimensional, continuous, or partially-observable multi-agent settings is an open question. The outer-loop operating conditions (noisy multi-seed evaluation, code-level edits to a real repository, bounded J-query budget) are realistic, but the conclusions about which strategies emerge are bounded by the inner-loop benchmark. Appendix B Additional Results Table 3: Strategies discovered by the researcher in Cleanup across 4 efficiency-optimized and 4 maximin-optimized independent runs. The grouped row records any explicit fairness mechanism; its two sub-rows decompose this into the dominant variant (time-based role rotation) and the alternative (synchronized whole-population clean/collect switching) found in one Sonnet maximin run. Strategy ΦU _U (4 runs) Φmin _ (4 runs) Waste-counting helpers 4/4 4/4 Zone/lane partitioning 3/4 4/4 Anti-regression feedback 3/4 2/4 Worked policy examples 2/4 3/4 Cleaning cost economics 2/4 3/4 Explicit fairness mechanism 0/4 4/4 of which: time-based role rotation 0/4 3/4 of which: synchronized clean/collect 0/4 1/4 Figure 3: Maximin (mini⁡Ri _iR_i) across researcher iterations for the 4 Cleanup maximin-optimized runs. All runs transform deeply negative baselines (worst-off agents losing reward) into substantially positive values. Gemini reaches ∼290 290 while Sonnet reaches ∼160 160–200200. The dashed line marks mini⁡Ri=0 _iR_i=0 (no agent loses reward overall). Figure 4: Final metrics across all conditions. (a) Efficiency: all conditions converge to U≈2.5U≈ 2.5–3.23.2. (b) Equality: maximin-optimized Cleanup runs achieve E≈1.0E≈ 1.0, while efficiency-optimized runs show E≈0.5E≈ 0.5–0.70.7; Gathering achieves high equality regardless. (c) Maximin: the sharpest contrast—efficiency optimization leaves the worst-off agent deeply negative (mini⁡Ri≈−200 _iR_i≈-200), while maximin optimization transforms it to +290+290 (Gemini) or +179+179 (Sonnet). Gathering achieves high maximin (∼580 580) under efficiency optimization alone. Error bars show s.d. across runs. Inner-loop averages confirm a broad-based improvement. Figures 2 and 3 report the metric of the kept inner-loop output (πc∗π^*_c) at each outer iteration. A natural concern is that this could overstate pipeline quality if the researcher is benefitting from variance: with K inner iterations per outer step, a single lucky generation could carry the curve while the rest of the inner trajectory is noise. Figures 5 and 6 replot the same runs with the metric averaged across all inner iterations of each outer step. The trajectories ramp at essentially the same rate, with only mild attenuation: Cleanup mean efficiency still climbs to U¯≈2.5 U≈ 2.5–3.03.0 (vs. 3.13.1–3.23.2 for the kept policy), Gathering still saturates at U¯≈2.5 U≈ 2.5, and Cleanup mean maximin still reaches mini⁡Ri¯≈200 _iR_i≈ 200–275275 (vs. 179179–290290). The whole inner-loop output distribution improves, not just its tail, consistent with the interpretation that the researcher is shaping the synthesizer’s behavior rather than amplifying lucky draws. Figure 5: Mean efficiency U¯ U averaged across all inner iterations of each outer step (compare Figure 2, which shows the kept policy only). Left: Cleanup (N=10N=10); right: Gathering (N=4N=4). The trajectories track Figure 2 closely, indicating that the researcher’s gains come from improving the entire inner-loop output distribution, not just the best of K samples. Figure 6: Mean maximin mini⁡Ri¯ _iR_i averaged across all inner iterations of each outer step for the 4 Cleanup Φmin _ runs (compare Figure 3). The deeply negative baselines lift to ∼ 200–275 mean maximin, only mildly below the kept-policy values of Figure 3. The horizontal dashed line marks mini⁡Ri¯=0 _iR_i=0. B.1 Discard Taxonomy Table 4: Discard taxonomy across 12 main-experiment runs (100 outer iterations). The four named-mode categories (iteration regression, over-prescription, feedback overload, variance probe) correspond to the failure patterns described in Section 4; “pure J-regression” captures iterations where ℛR’s edit underperformed the running best J∗J^* without an identifiable named cause. Category Count Share of discards Kept (Jj>J∗+τJ_j>J^*+τ) 47 — Baselines (unmodified pipeline, j=0j=0) 7 — Discarded (Jj≤J∗J_j≤ J^*) 46 100% pure J-regression (no identified failure mode) 20 43% iteration regression (e.g., K∈4,5K∈\4,5\ over-refinement) 11 24% over-prescription (verbose hints / restrictive worked examples) 8 17% feedback overload (verbose diagnostics, anti-regression guards) 4 9% variance probe (deliberate re-run at fixed c to gauge seed noise) 3 7% Total outer iterations logged 100 A natural question raised by Figure 2 is how much of the open-circle “discard” mass is genuine exploration that regresses on J versus iterations that fail for reasons unrelated to the objective. Aggregating across all 12 main-experiment runs (Table 4), ℛR ran 100 outer iterations: 7 baselines (the unmodified pipeline at j=0j=0 for each run, plus two re-baselines), 47 kept (Jj>J∗+τJ_j>J^*+τ with τ=0τ=0), and 46 discarded. No outer iteration was discarded for AST safety or smoke-test failure; those events occur inside the inner loop and are absorbed by the regeneration mechanism (up to R retries with the error message appended to the prompt, Section 3). All 46 discards are therefore regressions on J; we further classify them by which of the failure modes in Section 4 (“Common failure modes”) we can identify from the researcher’s logged description and the per-iteration metric trajectory. Three observations follow. First, no discards are generation/validation failures at the outer loop, confirming that the AST-and-smoke-test guard plus regeneration-on-error is a non-leaky boundary between inner-loop code-validity issues and outer-loop strategy decisions. Second, the named failure modes from Section 4 account for 23/4623/46 (50%50\%) of discards; the remaining 20/4620/46 (43%43\%) are “pure” regressions on J where ℛR tried an intuitively plausible modification (a new helper, a reworded hint, a different |S||S|) that simply underperformed the running best. Third, the researcher occasionally elects to spend an iteration on a deliberate variance probe (re-running the same c on a different seed set) — these constitute 3/463/46 of discards and reflect ℛR actively reasoning about evaluator noise rather than only chasing J. The breakdown by cell is qualitatively consistent: under ΦU _U on Cleanup, iteration regression (7/227/22) and feedback overload (4/224/22) dominate, matching the volatility of high-U Cleanup policies; under Φmin _ on Cleanup, pure J-regression dominates (12/2112/21), reflecting the broader plateau of competitive maximin configurations ℛR explores before locking in a rotation template; Gathering is sparser (33 discards across 44 runs), consistent with that game’s faster saturation. B.2 Spec-gaming inspection The configuration tuple c=(p,ϕ,ℋ,ι)c=(p,φ,H, ) is unconstrained: ℛR is free to edit the helper library ℋH and the feedback function ϕφ in ways that could in principle game the held-out evaluation J (e.g., by exposing simulator internals to ℳM, hard-coding actions, or overfitting hyperparameters to the eval seed set). We inspected the final pipelines from all runs for such patterns. We found no helpers that hard-code action sequences, manipulate the environment’s RNG, or expose information beyond what is in principle observable from the gridworld state: ℋH’s edits across runs are spatial heuristics (BFS-Voronoi, zone partitioning), state queries (waste_fraction, count_alive_apples_in_cols, apples_respawning_soon), and action-effect previews (best_clean_orientation simulates the beam geometry). ϕφ’s edits are thresholded interventions on the optimized metric itself (e.g., a “regression” guard when latest U<U< running-best U; a “problem” alert when maximin <0<0): these condition feedback on the primary signal ℛR is being scored on, and use thresholds tied to the metric’s natural scale rather than to seed-specific values. On the iteration logic side, ℛR tends to walk |S||S| upward (5→8→125→8→12) when chasing maximin, which reduces seed-noise rather than overfitting to a small seed set. The closest behavior to spec-gaming we observed is the over-prescription failure mode of Section 4 (writing such a detailed worked example that ℳM effectively copies it) which manifests as a regression on J, not an undeserved gain, because the copied policy is brittle on held-out seeds. B.3 Smaller Open-Weight Model: Gemma 4 26B We test the framework with Gemma 4 26B-A4B-IT (Google), a 26B-parameter open-weight model substantially smaller than the frontier models in the main experiments. We run three experiment runs: two on Cleanup optimizing efficiency and maximin respectively, and one on Gathering optimizing efficiency. As with the other synthetizer models, all with Opus 4.6 as the researcher ℛR. Setup. In all three experiments, Gemma starts from complete failure: the baseline pipeline produces U=−10.0U=-10.0 on Cleanup (agents CLEAN-spam without collecting apples) and U=0.0U=0.0 on Gathering (broken BFS calls), compared to U=1.93U=1.93/0.860.86 (Gemini/Sonnet on Cleanup) and U=2.04U=2.04/0.030.03 (Gemini/Sonnet on Gathering). The researcher ran Jmax=10J_ =10–1818 outer iterations per condition. Results. Table 5 presents the best configurations discovered. Under efficiency optimization, the researcher achieves U=0.87U=0.87, a dramatic recovery from the broken baseline but far below frontier models (U≈3.2U≈3.2). The researcher compensates for the model’s limited capability by reducing inner-loop iterations to K=1K=1 (avoiding the regression that plagued K≥2K≥2 runs) and providing highly structured worked examples with explicit cleaning-role assignment. Table 5: Autoresearch with Gemma 4 26B-A4B-IT. Single run per condition. Baseline: pipeline from [4]. Game Target U E mini⁡Ri _iR_i Keep rate Cleanup Baseline −-10.0 1.00† −-1000 — ΦU _U 0.87 −-1.11 −-371 6/19 Φmin _ 1.71 0.94 137 9/17 Gathering Baseline 0.0 1.00† 0 — ΦU _U 2.44 0.98 580 4/11 † is trivially 1.0 because all agents receive near-zero reward. Maximin optimization rescues efficiency. Strikingly, under maximin optimization the researcher achieves higher efficiency (U=1.71U=1.71) than under direct efficiency optimization (U=0.87U=0.87), alongside strong equality (E=0.94E=0.94) and positive worst-off welfare (mini⁡Ri=137 _iR_i=137). This reversal, absent in frontier models where both objectives yield U≈3.1U≈3.1–3.23.2, occurs because the maximin objective forces the researcher toward coordination mechanisms (rotating cleaning duties, index-based apple assignment) that simultaneously improve collective output. Under efficiency-only optimization, the researcher converges on a local optimum that a 26B model can implement but that caps performance well below the frontier. This suggests that for models below a capability threshold, fairness objectives may serve as better optimization targets for overall social welfare than direct efficiency maximization. The structured coordination enforced by the maximin objective provides a scaffold that compensates for the weaker model’s difficulty implementing complex strategies from strategic hints alone. Gathering: full recovery on a simpler game. On Gathering (N=4N=4), the researcher brings Gemma from U=0.0U=0.0 to U=2.44U=2.44 with E=0.98E=0.98 and mini⁡Ri=580 _iR_i=580, after 10 outer iterations (4 kept). These values nearly match frontier models (Gemini: U=2.49U=2.49, Sonnet: U=2.52U=2.52; Table 2). The researcher discovers the same Voronoi partitioning and respawn-aware camping strategies found in the main experiments, and increases inner-loop iterations to K=5K=5 to allow sufficient refinement. The contrast with Cleanup suggests that the researcher can fully compensate for model weakness on simpler coordination tasks (4 agents, symmetric costs) but not on harder provision dilemmas (10 agents, asymmetric costs). B.4 Compute Requirements Table 6 reports wall-clock time for all 12 autoresearch runs. Inner loop measures total time spent on policy LLM generation and environment simulation across all evaluations; total (estimated from run-directory timestamps) additionally includes the researcher agent’s analysis, code editing, and decision-making between evaluations. Table 6: Wall-clock time per autonomous run. Evals: total inner loop evaluations including initial baseline. Per eval: mean wall-clock per single evaluation. Total: end-to-end including researcher overhead. Policy LLM Φ Evals Inner loop (h) Per eval (min) Total (h) Cleanup (N=10N=10) Gemini ΦU _U 11 2.6 14 3.7 Gemini ΦU _U 18 4.1 14 6.4 Sonnet ΦU _U 4 2.1 32 6.1 Sonnet ΦU _U 7 5.0 43 6.1 Gemini Φmin _ 17 3.0 11 4.3 Gemini Φmin _ 11 3.6 20 5.0 Sonnet Φmin _ 8 4.5 33 8.8 Sonnet Φmin _ 9 5.3 36 13.2 Gathering (N=4N=4) Sonnet ΦU _U 3 1.7 33 2.2 Gemini ΦU _U 5 1.4 17 1.9 Gemini ΦU _U 3 0.9 19 1.1 Sonnet ΦU _U 4 2.3 35 3.4 All 12 runs 100 36.6 22 62.2 Cost breakdown. Each inner loop evaluation involves K=2K=2–33 policy LLM generation calls (each ∼ 2k input tokens, ∼ 1.5k output tokens) followed by multi-seed simulation (5–12 seeds, ∼ 15–30s total). Policy LLM generation dominates inner loop time (86–97%), with Sonnet evaluations taking ∼2× 2× longer than Gemini due to extended thinking. The researcher agent (Claude Opus 4.6, running via Claude Code CLI) accounts for 41% of total wall-clock time (25.6h of the 62.2h total across all 12 runs), spent reading results, editing pipeline source files, and planning modifications. In monetary terms, the researcher agent is the dominant cost: each outer iteration consumes ∼ 50–100k context tokens in the Opus session, whereas each inner loop evaluation uses only ∼ 5–10k policy LLM tokens. Appendix C Researcher-Authored Pipeline Artifacts The previous appendix shows the policies πc∗π^*_c that the inner loop outputs; this one shows the configuration c=(p,ϕ,ℋ,ι)c=(p,φ,H, ) (Section 4, Table 1) that the researcher ℛR authored to produce them. Excerpts are taken from the final commit on the dedicated git branch of the corresponding run; we reproduce the prose verbatim, with author commentary in [brackets] and elisions marked .... We organize by artifact type so each subsection makes a within-type contrast (e.g., ΦU _U vs. Φmin _ , or Cleanup vs. Gathering). Subsection C.1 comes first because it shows the upstream input that the rest of the appendix is derived from: the prompt pℛp_R given to the researcher itself. C.1 Researcher system prompt pℛp_R The researcher ℛR is a coding agent (Claude Opus 4.6 via the Claude Code CLI) instantiated with a single fixed system prompt that is identical across all experiment runs (it differs only in the game name and the primary-metric flag, which is one of efficiency or maximin). This is the input from which every artifact in the rest of this appendix is derived. Because much of the paper’s claim hinges on the asymmetry between what pℛp_R tells ℛR and what ℛR in turn writes for ℳM, we reproduce the strategically relevant portions of pℛp_R verbatim below; the full file is in the released repository under autoresearch/program.md. Listing 1: Excerpt from pℛp_R. The remaining parts of pℛp_R describe the file layout of the inner-loop pipeline, the evaluation script invocation, the keep/discard rule, and the logging format; none of those add task-specific strategic guidance. ⬇ ## The metrics The primary metric is specified at launch via ‘--metric‘ (default: ‘efficiency‘). The two options are: Efficiency (U): collective apple collection rate across all agents per timestep. Higher is better. ... Maximin (Rawlsian welfare): minimum total per-agent return across all agents. Higher is better. Inspired by Rawls’ difference principle -- a just policy maximizes the welfare of the worst-off agent. ... ## Strategy space Here are categories of modifications to explore: ### Prompt engineering (p) - Add strategic hints about the Cleanup dilemma (e.g., "cleaning is a public good -- someone must do it") - Add worked examples of sophisticated policies - Restructure the API documentation for clarity - Add game-theoretic reasoning frameworks - Mention optimal strategies from the literature (Voronoi partitioning, role assignment) ### Feedback engineering (l, phi) - Show per-agent reward breakdown (not just average) - Add derived metrics (e.g., cleaning rate, waste level trends, apple growth) - Frame feedback to emphasize cooperation - Add temporal analysis ... - Show metric trends across iterations ... - Provide diagnostic hints based on metrics ... ### Helper functions (H) - count_waste(env), waste_fraction(env), bfs_to_waste(env, agent_id), should_clean(env), assign_role(env, agent_id), find_cleaning_position(env, agent_id) ### Iteration logic (iota) - Change K (more iterations = more refinement but more cost) - Change eval seeds, retry budget, thinking budget What pℛp_R does not say. For the convergent-discovery claim of Finding 4 to be informative, pℛp_R must not itself encode the specific mechanisms that ℛR later writes into p, ϕφ, ℋH under Φmin _ . Inspecting Listing 1, three absences matter: • No rotation formula. pℛp_R never mentions (agent_id + env._step_count // T) % n, env._step_count, or any other time-based cycling pattern. The closest item – “Mention optimal strategies from the literature (Voronoi partitioning, role assignment)” – names static role assignment, not time-rotated duty. • No objective-conditional guidance. The strategy-space listing is identical regardless of whether ℛR is launched with --metric efficiency or --metric maximin. pℛp_R does not tell ℛR to behave differently under the two objectives; the only objective-sensitive input is the (scalar) score JjJ_j returned after each inner-loop run. • No “rotation is for fairness.” pℛp_R does not link any strategy to either welfare criterion. The connection “rotation = fairness = high maximin” is constructed by ℛR from JjJ_j observations across iterations on the branch. The researcher-authored prompt p shown in Listing 4 and its Sonnet counterpart, both written under Φmin _ , contain all three of these elements; the corresponding ΦU _U prompts do not. The role of pℛp_R in the experiment is therefore to define the configuration space and the evaluation harness, not to script the discoveries themselves. C.2 Helper library ℋH: coordination primitives The researcher adds primitives that the policy LLM can call as black boxes, sparing it from re-implementing tricky logic on every iteration. Two patterns dominate: (i) state inspection (waste counts, fractions, beam-yield scoring) and (i) coordination primitives (zone assignment, role rotation). We show one of each. Listing 2: Cleanup, Φmin _ – band-based apple zoning helper added by the researcher in exp5. This is the primitive that lets the policy in Listing 7 keep collectors within their own row band, preventing all 7 gatherers from racing to the same apple. ⬇ 1def get_my_apples(env, agent_id): 2 """Alive apples in this agent’s horizontal row band. 3 Divides the orchard into n_agents bands; falls back to all apples if empty.""" 4 aid = int(agent_id) 5 n = int(env.n_agents) 6 band_h = env.height / n 7 band_lo, band_hi = aid * band_h, (aid + 1) * band_h 8 9 my_apples, all_apples = set(), set() 10 for i in range(env.n_apples): 11 if env.apple_alive[i]: 12 ar = int(env._apple_pos[i, 0]); ac = int(env._apple_pos[i, 1]) 13 all_apples.add((ar, ac)) 14 if band_lo <= ar < band_hi: 15 my_apples.add((ar, ac)) 16 return my_apples if my_apples else all_apples Listing 3: Gathering – BFS-Voronoi territory + respawn-aware waiting helpers added by the researcher in gather-exp3. These are the primitives that Listing 9 composes into a one-line policy: “go to my_zone_apples, otherwise nearest_respawning_apple.” ⬇ 1def voronoi_zones(env): 2 """Multi-source BFS Voronoi over walkable cells. 3 Returns (row, col) -> agent_id; ties broken by lower agent_id.""" 4 queue, visited = deque(), 5 for a_id in range(env.n_agents): 6 if int(env.agent_timeout[a_id]) == 0: 7 ar = int(env.agent_pos[a_id][0]); ac = int(env.agent_pos[a_id][1]) 8 queue.append((ar, ac, a_id)) 9 visited[(ar, ac)] = a_id 10 while queue: 11 r, c, a_id = queue.popleft() 12 for dr, dc in [(-1,0),(1,0),(0,-1),(0,1)]: 13 nr, nc = r + dr, c + dc 14 if 0 <= nr < env.height and 0 <= nc < env.width \ 15 and not env.walls[nr, nc] and (nr, nc) not in visited: 16 visited[(nr, nc)] = a_id 17 queue.append((nr, nc, a_id)) 18 return visited 19 20def nearest_respawning_apple(env, agent_id, zones=None, max_timer=10): 21 """Nearest dead apple in MY zone respawning within max_timer steps. 22 Returns (row, col) of best wait spot, or None.""" 23 if zones is None: zones = voronoi_zones(env) 24 best_pos, best_t, best_d = None, max_timer + 1, float(’inf’) 25 ar = int(env.agent_pos[agent_id][0]); ac = int(env.agent_pos[agent_id][1]) 26 for i in range(env.n_apples): 27 if not env.apple_alive[i]: 28 pos = (int(env._apple_pos[i][0]), int(env._apple_pos[i][1])) 29 if zones.get(pos) == agent_id: 30 t = int(env.apple_timer[i]) 31 if t <= max_timer: 32 d = abs(pos[0] - ar) + abs(pos[1] - ac) 33 if t < best_t or (t == best_t and d < best_d): 34 best_pos, best_t, best_d = pos, t, d 35 return best_pos The choice of helper depends on the dilemma’s geometry: row-bands suffice when fairness is the main concern (Cleanup maximin) but full BFS-Voronoi is needed when wall-aware territory ownership matters (Gathering). The researcher does not fix this in advance, it picks the simpler primitive whenever it works. C.3 System prompt p: from neutral framing to strategic briefing The unmodified system prompt is a neutral API description: “Write a policy that maximizes per-agent reward.” Under maximin optimization, the researcher rewrites the prompt into a multi-section strategic briefing. The excerpt below shows the pieces it added to the Cleanup prompt in exp5; the full prompt grew from 165 to 325 lines. Listing 4: Cleanup, Φmin _ – excerpts from the researcher-rewritten system prompt (exp5). The unmodified baseline contained only API documentation; everything below was added by ℛR. ⬇ ## CRITICAL OBJECTIVE: Rawlsian Fairness (Maximin) Your goal is to maximize the minimum per-agent total return across all agents. This is the "maximin" or Rawlsian welfare criterion. This means: it is NOT enough for the *average* agent to do well. The worst-off agent must do as well as possible. A policy where 8 agents each earn +200 but 2 agents each earn -100 is TERRIBLE (maximin = -100). Key implication: cleaning costs (-1 per CLEAN action) must be shared equitably among ALL agents. If you assign fixed "cleaner" roles, those agents will accumulate large negative rewards and destroy your maximin score. ## Strategy for Maximin: ROLE ROTATION + APPLE ZONING The optimal strategy for maximin involves: 1. Shared cleaning duty: ALL agents take turns cleaning using a rotation schedule based on ‘agent_id‘ and ‘env._step_count‘. For example: ‘is_my_cleaning_turn = (agent_id + env._step_count // SHIFT) % env.n_agents < NUM_CLEANERS‘ where SHIFT is ~50 steps and NUM_CLEANERS is 2-3. 2. When NOT your turn: collect apples from YOUR ZONE using ‘get_my_apples(env, agent_id)‘ -- prevents all gatherers competing for the same nearest apple. 3. NEVER use BEAM (action 6): it costs -1 and causes -50 to the target. 4. Keep waste density low: aim for ~2-3 active cleaners at any time. [... API documentation, helper documentation, working example ...] IMPORTANT: - NEVER assign permanent cleaning roles to specific agents -- this kills maximin. - Use env._step_count for time-based role rotation so all agents share cleaning duty. - NEVER use BEAM (action 6) -- it destroys both agents’ rewards. Two things stand out. First, the researcher writes the formula it expects ℳM to use almost verbatim ( (agent_id + env._step_count // SHIFT) % env.n_agents < NUM_CLEANERS ) and the policy in Listing 7 uses essentially this template. Second, the researcher teaches by counter-example: the explicit “8 agents earn +200, 2 earn -100, maximin = -100” makes the failure mode of ΦU _U-style static roles (Listing 6) concretely visible to ℳM. This is the information-design move predicted by the mechanism-design framing in Section 3.4. The Gathering prompt evolves along a different axis (no maximin, no rotation). Its researcher-added “Key Strategic Insights” section instead emphasizes (i) self-play implies never beam and (i) Voronoi partition each step, exactly matching what the policy in Listing 9 implements. C.4 Feedback ϕφ: adaptive diagnostics The baseline feedback function from [4] shows reward + four social metrics with definitions and stops there. The researcher turns it into a state machine: it inspects the latest metrics and conditionally injects different hints. Two different runs produced two qualitatively different diagnostics: the same metric (r¯k r_k or mini⁡Ri _iR_i) is repurposed to either stabilize a working policy or redirect a failing one. Listing 5: Adaptive feedback diagnostics. Top: efficiency-optimized run (exp1) – a stability guard prevents iter-3 regression once U is high. Bottom: maximin-optimized run (exp5) – two fairness diagnostics that fire when the worst-off agent loses reward. ⬇ 1# --- Cleanup, Phi_U feedback (exp1, pipeline/feedback.py final) --- 2last_eff = history[-1]["metrics"]["efficiency"] 3if last_eff >= 2.5: 4 parts.append( 5 "**CRITICAL -- DO NOT REGRESS**: The current policy achieves high " 6 "efficiency (>=2.5). You MUST output a policy that is nearly identical " 7 "to the current one. Copy the current policy code and make AT MOST " 8 "one small targeted improvement (e.g., adjust a single numeric " 9 "threshold). If you are not confident a change will help, output the " 10 "current policy UNCHANGED. A regression here means the run fails.") 11 12# --- Cleanup, Phi_min feedback (exp5, pipeline/feedback.py final) --- 13last_maximin = history[-1]["metrics"].get("maximin", 0) 14last_avg = history[-1]["reward_avg"] 15if last_maximin < 0: 16 parts.append( 17 f"**FAIRNESS ALERT**: maximin=last_maximin:.1f is NEGATIVE. " 18 f"The worst-off agent lost reward overall while average was last_avg:.1f. " 19 "This means cleaning duties are NOT shared equitably. " 20 "Use time-based role rotation (env._step_count) so ALL agents share " 21 "cleaning costs equally. NEVER assign permanent cleaner roles.") 22elif last_maximin < last_avg * 0.5: 23 parts.append( 24 f"**FAIRNESS WARNING**: maximin=last_maximin:.1f is much lower than " 25 f"average=last_avg:.1f. The gap suggests unequal cleaning burden. " 26 "Ensure ALL agents rotate through cleaning duty.") The two diagnostics are written by independent researcher runs but converge on a common pattern: thresholded interventions on a primary metric. They are also a common cause of the low run-to-run variance reported in Finding 1: when a working policy lands in the high-U basin, the stability guard prevents the LLM from over-refining and tipping out of it (the K=3K=3 regressions visible in Section 4’s “Common failure modes” (4)), and when an unfair policy lands in the negative-maximin region, the alert injects exactly the rotation hint that recovers it. C.5 Iteration logic ι : per-condition hyperparameters Although ι has the smallest source surface, the researcher’s edits to it explain a meaningful fraction of the variance reduction noted in Finding 1. Table 7 summarizes the values shipped in the final commit of each best-performing run. Table 7: Iteration-logic (ι ) values in the final commit of selected runs. K = inner-loop iterations, |S||S| = evaluation seeds, “thinking” = extended-thinking token budget passed to ℳM. Baseline ([4]) is K=3K=3, |S|=5|S|=5, thinking 1616k. Run Game / objective K |S||S| thinking Researcher’s stated rationale exp1 (Gemini) Cleanup, ΦU _U 3 5 16k default; stability comes from feedback hint exp4 (Sonnet) Cleanup, ΦU _U 3 5 32k “Sonnet was over-refining at 16k thinking” exp5 (Gemini) Cleanup, Φmin _ 2 12 16k “K=2K=2 avoids iter-3 regression; 12 seeds for variance control” exp7 (Sonnet) Cleanup, Φmin _ 3 5 10k “reduced thinking from 16k – Sonnet was generating runaway code at 16k” gather-exp3 (Gemini) Gathering, ΦU _U 3 5 16k default; Gathering converges in K≤3K≤3 The maximin Gemini run is the most informative case: the researcher discovered that with K=3K=3 and |S|=5|S|=5, identical configurations could yield mini⁡Ri=295 _iR_i=295 on one set of seeds and mini⁡Ri=100 _iR_i=100 on another (this is the “variance exposure” failure mode of Section 4’s common failures). It then walked |S||S| from 5→8→125→ 8→ 12 over three outer iterations until the seed-averaged signal was stable enough that the policy LLM stopped chasing noise. The policy in Listing 7 is sampled at |S|=12|S|=12. C.6 Cross-artifact observations • Helpers do the algebra; prompts pick the strategy class. The researcher uses helpers (2, 3) to put non-trivial primitives at ℳM’s fingertips, and the prompt (4) to tell ℳM which primitive to compose. Prompts without supporting helpers produced policies that “mention rotation” but failed to implement it; helpers without prompt updates often went unused. • Feedback is the only adaptive component. p, ℋH, and ι are static within a run; ϕφ is the only place where the researcher gets to change what ℳM sees during the inner loop, and it uses thresholded diagnostics (Listing 5) to do so. This is what mostly drives the run-to-run variance reduction. • The same artifact has different content under each objective. The Cleanup prompt under ΦU _U omits “rotation” and “maximin” entirely; the same prompt under Φmin _ devotes its entire opening section to it. The configuration c is genuinely a function of the welfare objective Φ , not a fixed pipeline parameterized by it. This is consistent with the information-design framing of Section 3.4. • The researcher’s edits are short and targeted. The full diff between the baseline pipeline and the best maximin Gemini configuration totals ∼300 300 added lines spread over four files; ∼80 80 of them appear in the listings above. The remaining content is API documentation and worked examples, included for completeness but not strategically novel. Appendix D Selected Generated Policies The code excerpts below are verbatim outputs of the policy synthesizer ℳM, taken from the final inner-loop iteration of the best-performing run in each condition. To fit the page, we elide repeated boilerplate (BFS scaffolding, fallback branches) with ... and add commentary in [brackets]; nothing else has been edited. The listings illustrate the qualitative differences highlighted in Findings 2–4 of Section 4: static vs. rotating cleaning roles in Cleanup, and the Voronoi + respawn-timer structure in Gathering. All four policies were independently generated by the LLM under different seeds, models, and outer-loop runs. D.1 Cleanup, ΦU _U (Gemini): static interleaved roles + dynamic threshold Best run: U=3.25U=3.25, E=0.61E=0.61, mini⁡Ri=−182 _iR_i=-182. The synthesizer assigns permanent cleaning roles via agent_idmodn<τ agent\_id n<τ, where the threshold τ depends smoothly on the current waste fraction. Higher pollution recruits more cleaners, but the identity of each cleaner is fixed: agents 0–τ−1τ-1 always pay the cost. This drives U to its ceiling but leaves equality at 0.610.61 and pushes the worst-off agent to mini⁡Ri=−182 _iR_i=-182. Listing 6: Cleanup, ΦU _U – Gemini 3.1 Pro. Static role assignment with waste-tiered threshold (verbatim, abridged). ⬇ 1def policy(env, agent_id) -> int: 2 if int(env.agent_timeout[agent_id]) > 0: 3 return 7 # STAND while removed 4 5 waste_frac = waste_fraction(env) 6 n_agents = env.n_agents 7 8 # [Smooth dynamic threshold: more pollution -> more cleaners] 9 if waste_frac > 0.35: threshold_cleaners = n_agents 10 elif waste_frac > 0.25: threshold_cleaners = int(n_agents * 0.7) 11 elif waste_frac > 0.15: threshold_cleaners = int(n_agents * 0.5) 12 elif waste_frac > 0.05: threshold_cleaners = int(n_agents * 0.3) 13 else: threshold_cleaners = max(1, int(n_agents * 0.2)) 14 15 # [STATIC role: cleaners are ALWAYS the lowest-id agents -- no rotation] 16 is_cleaner = (agent_id % n_agents) < threshold_cleaners 17 18 if is_cleaner: 19 best_orient, n_waste = find_best_clean_orientation(env, agent_id) 20 if n_waste >= 1: 21 cur_orient = int(env.agent_orient[agent_id]) 22 if best_orient == cur_orient: return 8 # CLEAN 23 if (cur_orient + 1) % 4 == best_orient: return 5 # ROTATE_RIGHT 24 elif (cur_orient - 1) % 4 == best_orient: return 4 # ROTATE_LEFT 25 else: return 4 26 # [Competitor-aware BFS toward waste in own row band] 27 c_idx = agent_id % n_agents 28 target_r = int(((c_idx + 0.5) / threshold_cleaners) * env.height) 29 # ... pick waste cell minimizing dist + |row - target_r| + 50 * #closer_cleaners 30 # ... bfs_toward and direction_to_action 31 else: 32 # GATHERER: row-banded sweep, competitor-aware 33 # assigned_row = ((g_idx + 0.5) / n_gatherers) * env.height 34 # ... pick alive apple minimizing dist + |row - assigned_row| + 50 * #closer_gatherers 35 ... D.2 Cleanup, Φmin _ (Gemini): time-rotated, geographically interleaved roles Best run: U=3.19U=3.19, E=0.98E=0.98, mini⁡Ri=296 _iR_i=296. Switching the outer objective from U to mini⁡Ri _iR_i flips the role-assignment scheme entirely: role_idx now depends on agent_id+step/50 agent\_id+ step/50, so every agent rotates through cleaner and gatherer duty in a 50-step cycle. The cleaner indices 1,4,8\1,4,8\ are spatially interleaved along the river (rather than contiguous) so that whoever is on duty has a short walk to their assigned river slice. This single change accounts for the E=0.55→0.98E=0.55→ 0.98 jump and the mini⁡Ri=−182→296 _iR_i=-182→ 296 swing reported in Table 2, with negligible efficiency loss. Listing 7: Cleanup, Φmin _ – Gemini 3.1 Pro. Time-rotated roles with interleaved geography (verbatim, abridged). ⬇ 1def policy(env, agent_id) -> int: 2 if int(env.agent_timeout[agent_id]) > 0: 3 return 7 # STAND while removed 4 5 aid = int(agent_id) 6 step = int(env._step_count) 7 n = int(env.n_agents) 8 9 # [TIME ROTATION: every 50 steps each agent’s role index advances] 10 if n == 10: 11 role_idx = (aid + step // 50) % 10 12 # [Cleaner indices 1,4,8 are SPATIALLY INTERLEAVED, not contiguous, 13 # so each on-duty cleaner has a short walk to its river slice] 14 if role_idx == 1: role_type, zone_idx = ’C’, 0 15 elif role_idx == 4: role_type, zone_idx = ’C’, 1 16 elif role_idx == 8: role_type, zone_idx = ’C’, 2 17 else: 18 role_type = ’G’ 19 g_map = 0:0, 2:1, 3:2, 5:3, 6:4, 7:5, 9:6 20 zone_idx = g_map[role_idx] 21 num_cleaners, num_gatherers = 3, 7 22 23 if role_type == ’C’: 24 # [Each on-duty cleaner owns one horizontal slice of the river] 25 row_min = int( zone_idx * env.height / num_cleaners) 26 row_max = int((zone_idx + 1) * env.height / num_cleaners) - 1 27 28 # [Score every (row, col, orientation) in the slice by beam yield; 29 # fire if facing >=2 waste, else move to a >=2-yield position] 30 best_positions, good_positions = set(), set() 31 for r in range(row_min, row_max + 1): 32 for c in range(10): 33 if env.walls[r, c] or env.waste[r, c]: continue 34 y_max = max(get_clean_yield(env, r, c, o) for o in range(4)) 35 if y_max >= 2: best_positions.add((r, c)) 36 if y_max >= 1: good_positions.add((r, c)) 37 # ... rotate / CLEAN / bfs_to_target_set as appropriate 38 else: 39 # GATHERER: collect apples ONLY inside the rotating row band 40 # row_min/row_max for num_gatherers slices ... 41 ... D.3 Cleanup, Φmin _ (Sonnet): independent rediscovery of duty rotation This run was launched on a separate dedicated git branch from the Gemini maximin runs of Listing 7 and from pℛp_R identical to the one shown in Appendix C.1, with no shared state. The researcher-authored synthesizer prompt p on this branch ended up containing a rotation hint structurally similar to Listing 4 but with different phrasing and a different recommended period (phase_length= 100\,=\,100 in a worked example, vs. SHIFT≈ 50\,≈\,50 in Listing 4); ℳM then chose its own period of 5050. The convergence is therefore at the level of which artifacts ℛR injects into p, not at the level of ℳM inventing rotation from a neutral prompt. Best run: U=2.93U=2.93, E=0.83E=0.83, mini⁡Ri=154 _iR_i=154. The Sonnet synthesizer arrives at the same structural insight as the Gemini maximin run: a phase counter (agent_id+step/50)modn( agent\_id+ step/50) n rotates which 2 of the 10 agents clean at any given time, and a separate 200-step zone counter rotates which 5-row band each collector sweeps. The two rotation periods (50 and 200 steps) ensure every agent visits every cleaning slot and every apple zone within one episode, resulting in a structural fairness invariant. An alternative Sonnet maximin run uses a different mechanism. The second Sonnet Φmin _ run took a structurally distinct route. There, ℛR never wrote a rotation template into p; instead it authored a “collective threshold” worked example in which all agents synchronously enter a cleaning mode whenever waste_fraction(env)> 0.22\,>\,0.22 and a collecting mode when it drops below 0.080.08, with no agent_id phase shift. The resulting synthesizer policy follows this pattern and reaches mini⁡Ri=200 _iR_i=200, slightly above the rotation policy of Listing 8, confirming that the convergence in Table 3 is on the broader class (explicit fairness mechanism, 4/4 maximin) rather than on rotation specifically. Listing 8: Cleanup, Φmin _ – Sonnet 4.6 (rotation variant; see Appendix D.3 for the alternative synchronized clean/collect mechanism in the second Sonnet maximin run). Phase-rotated cleaning + zone-rotated collection (verbatim, abridged). ⬇ 1def policy(env, agent_id) -> int: 2 if int(env.agent_timeout[agent_id]) > 0: 3 return 7 # STAND while removed 4 5 n = env.n_agents 6 step = env._step_count 7 wf = waste_fraction(env) 8 9 # --- ROLE ASSIGNMENT: fair 2-cleaner rotation --- 10 # [50-step phases x 20 phases / episode -> each agent cleans 4 phases (~20%)] 11 phase = step // 50 12 cleaner_rank = (agent_id + phase) % n # [cycles 0..9 fairly] 13 is_cleaner = cleaner_rank < 2 # [exactly 2 cleaners per phase] 14 15 # [Emergency override: all agents clean when waste crosses the spawn cliff] 16 in_emergency = wf >= 0.40 17 if in_emergency: is_cleaner = True 18 19 # [River split: cleaner_slot 0 -> top half, 1 -> bottom half] 20 cleaner_slot = (agent_id % 2) if in_emergency else cleaner_rank 21 22 if is_cleaner and wf > 0.04: 23 # [Scan 4 orientations from current position; if best beam shot 24 # covers >=3 waste cells, fire; else step to a strictly better neighbor] 25 # ... (best_o, best_cnt) = max over 4 directions 26 # ... if best_cnt >= 3: rotate to best_o, then CLEAN 27 # ... else: scan 4 adjacent cells x 4 orientations for higher yield 28 # ... if no waste in beam range: bfs to waste_set in OWN river half, 29 # fall back to whole river 30 31 # --- COLLECTOR LOGIC: 5-zone rotating apple collection --- 32 # [5 zones x 200 steps = 1000 steps -> each agent visits every zone once; 33 # 2 agents share each zone at any time, reducing competition 10 -> 2] 34 bounds = zone_boundaries(env, 5) 35 zone_phase = step // 200 36 zone = (agent_id + zone_phase) % 5 37 z_start, z_end = bounds[zone], bounds[zone + 1] 38 39 zone_apples = (int(env._apple_pos[i][0]), int(env._apple_pos[i][1])) 40 for i in range(env.n_apples) if env.apple_alive[i] 41 and z_start <= int(env._apple_pos[i][0]) < z_end 42 # ... bfs_to_target_set(zone_apples), else bfs_nearest_apple as fallback 43 return 7 D.4 Gathering (Gemini): wall-aware Voronoi + spatiotemporal targeting Best run: U=2.47U=2.47, E=0.98E=0.98, mini⁡Ri=580 _iR_i=580. In Gathering, where costs are symmetric, no role differentiation is needed: the entire optimization is spatial and temporal. The synthesizer (i) partitions cells into BFS-Voronoi territories owned by individual agents, (i) runs a single centralized BFS from its own position to score every reachable cell with exact wall-aware distances, and (i) for both alive and respawning apples in its territory, computes the earliest collection time max⁡(walk_dist,respawn_timer) ( walk\_dist,\, respawn\_timer) and targets the minimum. When the agent must wait on a dead apple, it steps onto a non-spawn adjacent cell rather than blocking a respawn point. This single sort key (max⁡(distance,timer) ( distance, timer)) subsumes both “go to nearest apple” and “camp respawn” as special cases. Listing 9: Gathering – Gemini 3.1 Pro. Voronoi territories + spatiotemporal priority targeting (verbatim, abridged). ⬇ 1def policy(env, agent_id) -> int: 2 if int(env.agent_timeout[agent_id]) > 0: 3 return 7 4 5 r, c = int(env.agent_pos[agent_id][0]), int(env.agent_pos[agent_id][1]) 6 orient = int(env.agent_orient[agent_id]) 7 8 # [Wall-aware Voronoi over the gridworld; spawn_points = all apple cells] 9 zones = voronoi_zones(env) 10 spawn_points = (int(p[0]), int(p[1])) for p in env._apple_pos 11 12 # [Single centralized BFS: exact distance + first move to every reachable cell] 13 distances, first_moves = , 14 q = deque([(r, c, 0, None)]) 15 visited = (r, c) 16 while q: 17 cr, c, d, fm = q.popleft() 18 distances[(cr, c)] = d 19 first_moves[(cr, c)] = fm 20 for dr, dc in [(-1,0),(1,0),(0,-1),(0,1)]: 21 nr, nc = cr + dr, c + dc 22 if 0 <= nr < env.height and 0 <= nc < env.width and not env.walls[nr, nc]: 23 if (nr, nc) not in visited: 24 visited.add((nr, nc)) 25 q.append((nr, nc, d + 1, fm if fm is not None else (dr, dc))) 26 27 # [Both alive AND respawning apples in MY zone, with their respawn timer] 28 my_zone_spawns = [] 29 for i in range(env.n_apples): 30 pr, pc = int(env._apple_pos[i][0]), int(env._apple_pos[i][1]) 31 if zones.get((pr, pc)) == agent_id and (pr, pc) in distances: 32 my_zone_spawns.append((pr, pc, int(env.apple_timer[i]), distances[(pr, pc)])) 33 34 # [SPATIOTEMPORAL PRIORITY: earliest collection time first. 35 # score = max(walk_distance, respawn_timer); ties broken by sooner timer] 36 my_zone_spawns.sort(key=lambda x: (max(x[3], x[2]), x[2], x[3])) 37 38 for pr, pc, timer, dist in my_zone_spawns: 39 if timer == 0: # [Alive apple: walk straight to it] 40 if dist == 0: return 7 41 fm = first_moves[(pr, pc)] 42 return direction_to_action(fm[0], fm[1], orient) 43 else: # [Dead apple: camp on safe adjacent cell] 44 # ... pick nearest neighbor (nr,nc) that is NOT in spawn_points, 45 # navigate there and STAND while waiting for respawn 46 ... 47 # [Global poaching fallback if zone is empty: nearest alive apple anywhere] 48 ... D.5 Cross-condition observations Several patterns are visible across these four listings (and confirmed by inspecting the remaining 8 runs not shown): • Role assignment encodes the welfare objective. Static agent_id < τ (Listing 6) maximizes U but harms mini⁡Ri _iR_i. Time-rotated (agent_id + step//T) % n (Listings 7, 8) maximizes mini⁡Ri _iR_i at ≤1%≤ 1\% efficiency cost (Gemini). • Convergent rediscovery across LLMs. Gemini and Sonnet, on independent runs with separate dedicated git branches, both arrive at the (id + phase) % n duty-rotation idiom with similar phase lengths (T≈50T≈ 50 steps) in 3/4 maximin runs, and both omit any explicit fairness mechanism under efficiency. The remaining 1/4 maximin run (Sonnet) converges on a structurally distinct synchronized clean/collect mechanism with comparable maximin (Appendix D.3). The point of convergence is the class of mechanism (ℛR writes an explicit fairness mechanism into p under Φmin _ , never under ΦU _U), not necessarily the rotation idiom itself. • Game structure dictates strategy class. Cleanup policies are dominated by role-assignment logic (cleaner vs. gatherer); Gathering policies are dominated by territory partitioning and respawn-timer reasoning, with no role differentiation at all. The researcher does not need to be told which class of strategy to write. • Earliest-collection-time as a unifying score. The Gathering policy’s max⁡(walk,timer) (walk,timer) key (Listing 9) is structurally identical across the 4 Gathering runs (both Gemini and Sonnet), differing only in how the Voronoi diagram is computed, e.g. by Manhattan approximation, fully BFS-based, or cached helper.

Instant research discovery