Paper deep dive

Training Small LLMs as Spatial Multi-Agent Policies

Yi Mao, Andrew Perrault

Year: 2026Venue: arXiv preprintArea: cs.MAType: PreprintEmbeddings: 64

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%

Last extracted: 8/4/2026, 3:31:37 AM

Summary

This paper proposes a method for training small, frozen Large Language Models (LLMs) as policies in spatial multi-agent reinforcement learning (MARL) environments. The approach addresses three key obstacles: grounding gaps, horizon problems, and specialization issues. It introduces a symbolic option library drafted by a frontier coding model and filtered by mechanical feasibility guards derived from random-policy rollouts. Agents use private Low-Rank Adaptation (LoRA) adapters trained via a per-agent variant of Multi-Agent Group Relative Policy Optimization (PA-MAGRPO). The study demonstrates that this method lifts frozen models from zero reward to competent play across three games. Crucially, behavioral audits reveal that team reward and cooperation are decoupled; high reward does not necessarily imply cooperative behavior, as agents may learn to perform tasks alone while partners idle.

Entities (10)

LoRA · technique · 95%Amato, Konidaris, and Kaelbling 2014 · citation · 90%Claude Opus 4.8 · model · 90%Dec-POMDP · framework · 90%PA-MAGRPO · algorithm · 90%Qwen3-4B-Instruct-2507 · model · 90%Sutton, Precup, and Singh 1999 · citation · 90%Cleanup · environment · 85%MAGRPO · algorithm · 85%Overcooked-AI · environment · 85%

Relation Signals (8)

Reward → decouplesfrom → Cooperation

confidence 95% · Behavioral audits then reveal that reward and cooperation decouple: a rising reward curve may simply mean that one agent has learned to run the entire task alone while its partner idles

Claude Opus 4.8 → drafts → Option Library

confidence 90% · Each game’s option library is built with exactly one frontier-model step: Claude Opus 4.8 reads the game’s source code and drafts a typed option schema

Cleanup → exhibits → Mode Collapse

confidence 90% · Failure 2 — mode collapse on a “safe” default... the action distribution collapses to a single letter: fire_clean 95.8%

Cleanup → exhibits → Spatial Hallucination

confidence 90% · Failure 1 — spatial hallucination... The agent faces west; the only visible dirt is north of it... The model has the symbols and ignores their meaning.

PA-MAGRPO → uses → LoRA

confidence 90% · Each agent’s LLM acts as its policy over options, with a private per-agent LoRA adapter trained by a per-agent variant of multi-agent GRPO (PA-MAGRPO)

Symbolic Option Library → addresses → Horizon Problem

confidence 85% · Option-level decisions compress the horizon: the LLM is consulted on only a fraction of environment steps.

Symbolic Option Library → addresses → Grounding Gap

confidence 85% · The first two obstacles have a classic remedy: temporal abstraction... Instead of asking the LLM to steer with primitive commands... we let it choose among a small set of options

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Training LLM-based multi-agent systems with multi-agent reinforcement learning is rapidly gaining traction, and a parallel line of work argues that such systems should be judged by their behavior, not only their reward. We take up both threads in spatial cooperative games, where small frozen LLMs prompted with low-level actions fail outright, earning zero reward. Guided by the options/semi-MDP framework---and, because option execution is asynchronous across agents, its multi-agent extension in macro-action Dec-POMDPs---we equip each game with a library of symbolic \emph{options}: typed, state-feasible, short-horizon behaviors executed by a symbolic planner. Each library is drafted by a frontier coding model from the game's source code; the feasibility guards that filter each menu are then synthesized mechanically from cheap random-policy burn-in rollouts---a guard is adopted only if it explains repeated execution failures while hiding no logged success---so no guard is authored, selected, or reward-tuned by hand. Each agent's LLM acts as its policy over options, with a private per-agent LoRA adapter trained by a per-agent variant of multi-agent GRPO (PA-MAGRPO); this lifts frozen bases from zero reward to competent play across three games and four small backbones. Behavioral audits then reveal that reward and cooperation decouple: a rising reward curve may simply mean that one agent has learned to run the entire task alone while its partner idles---cooperation emerges only when the task makes it necessary. Reward alone is thus an unreliable readout of cooperation; behavioral evaluation must sit alongside it.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

62,825 characters extracted from source content.

Expand or collapse full text

Training Small LLMs as Spatial Multi-Agent Policies Yi Mao, Andrew Perrault Abstract Training LLM-based multi-agent systems with multi-agent reinforcement learning is rapidly gaining traction, and a parallel line of work argues that such systems should be judged by their behavior, not only their reward. We take up both threads in spatial cooperative games, where small frozen LLMs prompted with low-level actions fail outright, earning zero reward. Guided by the options/semi-MDP framework—and, because option execution is asynchronous across agents, its multi-agent extension in macro-action Dec-POMDPs—we equip each game with a library of symbolic options: typed, state-feasible, short-horizon behaviors executed by a symbolic planner. Each library is drafted by a frontier coding model from the game’s source code; the feasibility guards that filter each menu are then synthesized mechanically from cheap random-policy burn-in rollouts—a guard is adopted only if it explains repeated execution failures while hiding no logged success—so no guard is authored, selected, or reward-tuned by hand. Each agent’s LLM acts as its policy over options, with a private per-agent LoRA adapter trained by a per-agent variant of multi-agent GRPO (PA-MAGRPO); this lifts frozen bases from zero reward to competent play across three games and four small backbones. Behavioral audits then reveal that reward and cooperation decouple: a rising reward curve may simply mean that one agent has learned to run the entire task alone while its partner idles—cooperation emerges only when the task makes it necessary. Reward alone is thus an unreliable readout of cooperation; behavioral evaluation must sit alongside it. 1 Introduction Training LLM-based multi-agent systems with multi-agent reinforcement learning is rapidly gaining traction: recent work moves beyond prompting frozen frontier models (Mosquera et al. 2025; Gallego 2026) and optimizes the interacting agents themselves (Ma et al. 2024; Liu et al. 2026; Piche et al. 2025), while a parallel line of work argues that such systems should be judged by their behavior rather than by reward alone (Leibo et al. 2017). A frozen model can only be steered by prompting, so its coordination failures are permanent; a trainable policy can instead turn environment feedback into durable, agent-specific behavior. But joint RL multiplies rollout and gradient cost by the number of agents, pushing the trainable model size to small models (2–4B) under our academic budget. This paper asks what it takes to train small LLMs as spatial multi-agent policies—and what actually emerges when training succeeds. Three obstacles stand in the way. First, a grounding gap: in our experiments, small frozen models dropped into a 2D spatial game could not reliably turn a high-level goal (“clean the dirt to my north”) into the primitive actions the environment expects—they hallucinate spatial preconditions and collapse onto a single “safe” action (Valmeekam et al. 2023; Ahn et al. 2022; Huang et al. 2022; Liang et al. 2023)—so naive fine-tuning starts from, and never leaves, zero reward. Second, a horizon problem: episodes run for hundreds of primitive steps under a sparse team reward, leaving the optimizer almost no per-decision credit signal. Third, a specialization problem: cooperation often requires agents to behave differently, but a team trained through one shared set of weights averages conflicting role gradients and collapses onto identical behavior. A workable recipe has to address all three. The first two obstacles have a classic remedy: temporal abstraction. Instead of asking the LLM to steer with primitive commands (up, down, left, right, interact), we let it choose among a small set of options (Sutton, Precup, and Singh 1999). An option is a named, short-horizon behavior with a target—move_to(x,y) move\_to(x,y), pick_up(onion) pick\_up(onion), clean(nearest dirt) clean(nearest dirt). Once the agent selects move_to(x,y) move\_to(x,y), a symbolic executor computes a shortest-path route and emits the primitive moves: one decision spans multiple environment steps. Each option also carries checking code that tests, against the current state and observation, whether the option makes sense right now—and an option that fails its checks is simply not shown. If no apple is in the agent’s local view, eat does not appear in the prompt at all. At a decision point the LLM thus sees a short lettered menu of feasible options and simply picks one: it decides what to do, never how. Decisions become far fewer and semantically meaningful. Formally, each agent now faces a semi-MDP, and because options start and finish asynchronously across agents, the team-level problem is a macro-action Dec-POMDP (Amato, Konidaris, and Kaelbling 2014), in whose terms our method is decentralized macro-level policy learning (Xiao, Hoffman, and Amato 2020) with LLM policies. Each game’s option library is built with exactly one frontier-model step: Claude Opus 4.8 reads the game’s source code and drafts a typed option schema in which every option carries an effect predicate—a post-condition stating what the option promises to do. The feasibility guards that filter the menu are then synthesized by a fixed algorithm, with no model and no human in the loop: cheap uniform-random burn-in rollouts label every execution as valid or vacuous under the effect predicate, and a guard is adopted only if it explains repeated failures while hiding zero logged successes—a criterion of execution validity, never reward. After construction the frontier model is not queried again—not during training, evaluation, or action selection. The specialization obstacle motivates the second ingredient: a private LoRA adapter (Hu et al. 2022) per agent on a shared frozen base, trained jointly with a per-agent variant of multi-agent GRPO (Liu et al. 2026; Shao et al. 2024) (PA-MAGRPO): the shared base retains common language and task knowledge, while private adapters give divergent roles a parameter slot in which to emerge. Neither ingredient is redundant: the frozen base with the same option menus stays near zero reward on two of three games, and a single shared adapter collapses toward identical behavior. The recipe works: it lifts frozen bases from zero reward to competent play on all three games, and does so across four small backbones. We can also say why. Option-level decisions compress the horizon: the LLM is consulted on only a fraction of environment steps. The action space is semantic, where a language model’s priors help. And feasibility filtering removes the grounding failures outright. Training success makes the question visible: holding the recipe fixed, does a rising reward curve mean the team cooperates? Behavioral audits show the answer is: not necessarily. Across our three games, the same recipe yields high reward with genuine division of labor, high reward with one agent working while its partner idles, and nearly flat reward that hides a real shift to sustainable harvesting; coordination emerges only where the task demands it. Cooperation tracks the task’s incentive structure, not the reward curve. Contributions. 1. A symbolic option library that makes small LLMs viable spatial MARL policies. Each agent acts as a policy over a fixed library of symbolic options—the option menu interface: the LLM chooses what to do from a state-filtered menu, and a symbolic planner handles how. Formally, the interface recasts each agent’s problem as a semi-MDP over options and the team’s as a macro-action Dec-POMDP (Sutton, Precup, and Singh 1999; Amato, Konidaris, and Kaelbling 2014). Without it, neither the frozen base nor the fine-tuned policy produces meaningful behavior on the harder games. 2. Per-agent reinforcement fine-tuning (RFT) for multi-agent policies. We pair the option menu with per-agent LoRA adapters trained by MAGRPO (PA-MAGRPO). The shared base preserves common language and task knowledge, while private adapters give each agent a parameter slot where divergent behavior can emerge under a shared team return. This directly addresses the specialization obstacle above: it avoids the gradient-averaging failure of a single shared adapter and enables cleaner/eater specialization in Cleanup without explicit role labels. 3. Evidence that reward and cooperation decouple. Reward and cooperation decouple in both directions: Overcooked Asymmetric Advantages yields high reward from a one-worker/one-idler team (with Forced Coordination as the within-game control where deliveries do certify hand-offs), while Commons Harvest improves patch survival substantially under a nearly flat reward—reward curves can both overstate and understate cooperation. 2 Related Work Training LLM multi-agent systems. A growing line of work trains interacting LLM agents rather than prompting them: CORY (Ma et al. 2024) coevolves two copies of an LLM on text tasks; MAGRPO (Liu et al. 2026) introduces a group-relative multi-agent objective for collaborative writing and coding; Piche et al. (Piche et al. 2025) fine-tune open LLMs toward non-exploitable cooperation in textual social dilemmas; YOLO-MARL (Zhuang et al. 2025) keeps the LLM outside the gradient loop entirely, querying it once to synthesize a planner. We adopt MAGRPO’s group-relative objective but move it into embodied, partially observable spatial games, with per-agent adapters—a design established by LoRASA (Zhang, Kapoor, and Sun 2025) on SMAC and MAMuJoCo and by multi-LoRA coding pipelines (Lee, Cho, and Choi 2026). A second line argues that team reward alone is an insufficient signal of cooperation: sequential social dilemmas (Leibo et al. 2017) and common-pool resource models (Pérolat et al. 2017) measure behavior directly through social-outcome metrics (efficiency, equality, sustainability, peace); inequity aversion (Hughes et al. 2018) improves cooperation without altering the task reward; Biswas et al. (Biswas et al. 2026) show that MARL agents on Overcooked can attain high reward with very low interdependence; and Shapley-Coop (Hua et al. 2025) finds that self-interested LLM agents default to non-cooperation absent explicit credit mechanisms. These works establish, in single environments or single directions, that reward and cooperation can diverge; we join the two lines by training the agents ourselves and extending the divergence to a bidirectional, cross-task decoupling under one fixed recipe. Temporal abstraction and macro-action MARL. An option couples an initiation set, an intra-option policy, and a termination condition, and acting over a fixed option set turns an MDP into a semi-MDP (Sutton, Precup, and Singh 1999); with options terminating asynchronously across agents, the planning formalism is the macro-action Dec-POMDP (Amato, Konidaris, and Kaelbling 2014), over which macro-action-based deep MARL trains decentralized policies (Xiao, Hoffman, and Amato 2020). Our option menu instantiates this design with a frontier-drafted library where prior work typically hand-specifies one, and with a fine-tuned LLM rather than a task-specific network as the policy over options; unlike option-discovery methods, our options are fixed and symbolic, and learning happens only in the policy over them. Closest neighbors. Our design builds directly on three threads. Mosquera et al. (2025) show that Melting Pot cooperation can be posed to LLMs through a compound-action vocabulary (go_to(x,y), immobilize, stay_put, explore); we adopt essentially their vocabulary and ask what changes when the model can learn, rather than act as a frozen GPT-3.5/4 reasoner. Gallego (2026) shows that a frozen frontier model can author Python policies for social dilemmas under social-metric feedback; we likewise use a frontier model as a code author, but only at construction time—learning then happens inside the agents. Piche et al. (2025) demonstrate group-relative fine-tuning toward cooperation in textual social dilemmas such as Trust-and-Split; we bring that objective family onto spatial substrates with macro actions. Combining the three—spatial games, per-step small-LLM policies, and gradient-based multi-agent training—is to our knowledge new; the combination lets us hold one training recipe fixed across tasks and compare the behavior that emerges. Because these works differ in agent counts, horizons, and evaluation metrics, no direct numerical comparison is possible; we instead benchmark against frozen-base and ablation variants under an identical action interface, reporting behavioral metrics alongside return. 3 Background From Dec-POMDPs to macro-action Dec-POMDPs. The underlying game is a cooperative decentralized POMDP (Dec-POMDP) (Amato, Konidaris, and Kaelbling 2014) given by (,ii=1N,ii=1N,Ω,P,R,γ)(S,\O^i\_i=1^N,\A^i\_i=1^N, ,P,R,γ): at each step the environment is in state st∈s_t , each agent i receives a private observation oti=Ω(st,i)o^i_t= (s_t,i), the joint action t=(at1,…,atN)a_t=(a^1_t,…,a^N_t) drives the transition P(st+1∣st,t)P(s_t+1 s_t,a_t), and all agents share the team reward rt=R(st,t)r_t=R(s_t,a_t). The objective is a policy set =πii=1N π=\π^i\_i=1^N maximizing J()=[∑tγtrt]J( π)=E[ _tγ^tr_t]. Acting at the primitive level is not the only choice. An option z=⟨ℐz,πz,βz⟩z= _z, _z, _z couples an initiation set ℐzI_z (the observations in which z may be selected), an intra-option policy πz _z (here, a symbolic planner emitting primitive actions from iA^i), and a termination condition βz _z (here, completion, failure, or timeout of the macro) (Sutton, Precup, and Singh 1999). Acting over a fixed option set iZ^i turns agent i’s decision problem into a semi-MDP: decisions occur only at option boundaries, separated by a variable number of environment steps. We write i(oti)=z∈i:oti∈ℐzZ^i(o^i_t)=\z ^i:o^i_t _z\ for the menu of options available to agent i at observation otio^i_t; a policy over options πi(z∣oti)π^i(z o^i_t), supported on i(oti)Z^i(o^i_t), selects a new option whenever the previous one terminates. Because options begin and end asynchronously across agents, the team-level process is not itself a single semi-MDP; the resulting model—the Dec-POMDP above with each iA^i replaced by the option library iZ^i—is a macro-action Dec-POMDP (Amato, Konidaris, and Kaelbling 2014; Xiao, Hoffman, and Amato 2020): whenever agent i’s current option terminates, it selects a new option from its menu while its teammates continue executing theirs. Group-relative policy optimization. GRPO (Shao et al. 2024) replaces the learned value baseline of PPO-style methods with a cohort statistic: it samples K rollouts, normalizes their returns within the cohort, and uses the normalized return as the advantage in a clipped surrogate update with a KL penalty toward a reference policy. MAGRPO (Liu et al. 2026) extends this to cooperative multi-agent training: rollouts are joint, the scored quantity is the shared team return, and every agent’s update uses the same cohort-normalized advantage—avoiding a centralized critic, which is expensive and unstable at LLM scale. The Method section gives our per-agent instantiation. 4 Two Failure Modes of a Small LLM as a Cooperative Spatial Agent Before describing our recipe, we identify two failure modes of a naïve deployment on Cleanup: the frozen Qwen3-4B-Instruct-2507 base, no fine-tuning, is handed the standard prompt and asked to emit a primitive action letter A–I (movement, turning, fire_zap, fire_clean). Both failures are visible directly in the input/output stream. Failure 1 — spatial hallucination. The prompt tells the LLM its position, facing direction, nearest dirt, and a layout hint—everything needed to reason “to fire at dirt to my north, I must first face north.” In practice it does not. A verbatim trace from the first step of an episode: Prompt to agent 0 (truncated): You are agent 0. World position: (14,9). Facing: West. Sensors: Dirt visible 13, nearest (13,5) 5st, (15,5) 5st. Layout: SAND. River is to your north (rows 1--5). Response from agent 0: state: river_dirty choice: I ← FIRE_CLEAN The agent faces west; the only visible dirt is north of it; the beam discharges harmlessly into open sand. Across all five evaluation episodes (77 agents × 150150 steps), the LLM never once precedes a fire_clean with the turn_left/turn_right actions that the prompt itself enumerates as legal. The model has the symbols and ignores their meaning. We call this spatial hallucination: the LLM produces plausible-looking decisions whose pre-conditions (here, “facing north”) it has not verified. Failure 2 — mode collapse on a “safe” default. Even when no dirt is in view, the LLM emits the same letter: agents seeing Dirt visible: 0 still respond choice: I. Aggregated over all 5,250 LLM calls, the action distribution collapses to a single letter: fire_clean 95.8%95.8\%, noop 4.2%4.2\%, and the other seven primitives—including the turning actions that would let cleaning succeed—exactly 0%0\%. Worse, applying single-agent RL fine-tuning on the same prompt format does not unstick this attractor; it sharpens it: fire_clean share rises from 95.8%95.8\% to 98.4%98.4\% (Ablation A, §6). The cleaning bias lives in the base model’s prior over single-letter responses to a river_dirty prompt, and per-step letter-level RL only reinforces it. We call this mode collapse on a safe default: when the action interface forces a one-token-at-a-time micro-policy, our tested model picks the most semantically-loaded letter — the one whose macro-name evokes “cleaning” — and refuses to leave it. The two ingredients of our method target these two failures one-for-one; we describe them next and verify empirically that they fix the failures in the Experiments. 5 Method 5.1 A Symbolic Option Library and Its Menu Interface We lift the LLM’s decision space from primitive actions to the fixed library of symbolic options of §3, presented at each decision point as a lettered menu. The mapping onto the option formalism is exact: deterministic feasibility checks implement the initiation set ℐzI_z; a symbolic executor implements the intra-option policy πz _z, expanding a chosen option into primitive actions through shortest-path movement, orientation changes, and interaction preconditions; and completion, failure, or timeout of the macro implements termination βz _z. Feasibility is decided by code, not by a model—path reachability on the current map, held-item preconditions (placing an onion requires holding one), and task-state gates (plating requires a ready pot)—and an option that fails any check is simply not shown. The LLM, prompted with its local text observation and the surviving lettered menu, samples one option zti∈i(oti)z^i_t ^i(o^i_t). A Cleanup decision point renders as: Action menu: A) eat apple @ (2, 11) B) clean dirt @ (4, 5) C) go to the river D) go to the grass/orchard Response: choice: B (Schematic; a verbatim decision from the trained system appears in §5.1.) Departures from the strict options framework: our options are fixed and symbolic (no option discovery, no termination learning); execution is call-and-return (βz∈0,1 _z∈\0,1\, never interrupted); menu availability may consult state beyond the acting agent’s view and is computed centrally, while option selection and all learning remain per-agent; and the library is sampled per training seed rather than given as a single canonical set. How the option library is built. The pipeline has exactly one frontier-model step (Stage 1); Stages 2–4 are fixed algorithms with no model in the loop, and the guards are counted from execution data, never suggested by a model or selected by a human. Stage 1: option-schema drafting (the only model step) Claude Opus 4.8 (Anthropic) is prompted with the game’s source code and asked to produce a typed option library in which every option carries a four-part contract. The prompt is reproduced verbatim (Overcooked-AI, Asymmetric Advantages; the Cleanup and Commons Harvest prompts follow the same template with game-specific sections swapped): You are designing an option library for an LLM agent playing Overcooked-AI (layout: asymmetric_advantages, onion-only recipes). Produce a set of typed short-horizon options. For each option give a 4-part contract: 1. instantiation -- when does an instance appear in the menu (target existence only; NO state-conditional gating: do not encode game strategy or env legality rules -- those are unknown to you), 2. executor -- symbolic expansion into primitive actions (BFS to an interact pose + INTERACT). The executor may fail explicitly only for reasons internal to expansion (no target, no path, nothing to drop). 3. termination -- queue drained. 4. effect predicate -- a post-condition on the state delta that is TRUE iff the execution did what the option name promises. Do not add feasibility filters beyond what expansion itself requires; the guard set will be synthesized from execution data, not authored. The generated schema implements the contracts as code. The raw executor is deliberately environment-naive: def raw_expand(inst, snap, feat, aid, blocked=None): # Checks ONLY: target given, path exists, # drop has an item. if inst.target is None: return Expansion(failure= FailureReason("TargetAbsent", inst.opt)) if inst.opt == "drop_held" and held is None: return Expansion(failure= FailureReason("NothingHeld")) plan = shortest_interact_plan(inst.target, pos, orient, feat.walkable, blocked) if plan is None: return Expansion(failure= FailureReason("NoPath", str(inst.target))) return Expansion(actions=plan) These explicit checks are not authored gates in disguise: they are algorithmic preconditions of expansion itself—the domain of the partial function πz _z (path planning needs a target; a drop sequence needs an item)—and the prompt whitelists exactly these reasons. They hide nothing from the menu: the option still appears, the LLM can still select it, and the failure is simply logged. An executor failure becomes a menu gate only through Stage 3, by being counted in burn-in data like any other failure. Environment-legality knowledge (a pot must be non-empty and idle to start cooking) never appears in the executor; it is exactly what Stage 3 must discover. Effect predicates are written in delta form—the repair loop below explains why this is load-bearing—and shared-state effects are tied to a private transition of the acting agent: # acquire effects: delta form, not absolute if inst.opt == "pick_dish": return post["held"] == "dish" \ and pre["held"] != "dish" # shared-state effect bound to the PRIVATE held # transition, so a partner’s placement is never # attributed to this agent if inst.opt == "place_in_pot": return pre["held"] in ("onion", "tomato") \ and post["held"] is None \ and pot_count_increased(pre, post) Stage 2: uniform-over-menu burn-in (no LLM) A fixed harness runs episodes under a one-line policy—uniform random over the instantiated, guardless menu: inst = (policy(menu, snap, aid, rng) if policy else rng.choice(menu)) Each completed option execution emits one record: the option, its target, the declared read-set features at initiation, and an outcome in success, effect_fail, explicit:<reason>, decided by the effect predicate or the executor’s typed failure, never by reward: "ep": 512, "step": 41, "aid": 0, "opt": "start_cooking", "target": [4, 2], "features": "held": "onion", "tgt_pot_state": "full", "outcome": "effect_fail" The declared read set contains only deterministic features (held, tgt_pot_state, tgt_item); transient multi-agent features (e.g., partner adjacency) are logged but excluded from guard eligibility a priori. On Overcooked Asymmetric Advantages, 2,0002,000 solo plus 2,0002,000 two-agent episodes (∼550 550k decisions) take about 7070 seconds on one CPU core; no GPU is used in Stages 2–4. A control experiment replacing the uniform policy with a frozen pretrained LLM collapses coverage—its narrow deterministic attractor never reaches dish/plate/deliver states—yielding 22 guards versus uniform’s 3636 at roughly 400×400× the cost, which validates the uniform-burn-in choice. Stage 3: guard synthesis and the acceptance certificate Synthesis runs in two channels over the burn-in log. Explicit: a typed failure reason observed at least five times becomes a guard directly (NoPath → a live reachability check at menu build; NothingHeld → drop requires a held item). Inferred: every 1- and 2-literal conjunction over the read set is scored and adopted iff it covers at least five failures and zero logged successes: for conj in cands: # 1- and 2-literal conjs n_fail = sum(covers(conj, r) for r in fails) n_succ = sum(covers(conj, r) for r in succs) if n_fail >= MIN_SUPPORT and n_succ == 0: adopted.append(conj) # certificate: # hides ZERO successes The purity requirement is the acceptance certificate, and it is what keeps strategy out of the guard set: any conjunction that also covers a success—including successes of temporally extended executions whose precondition became true mid-walk—is rejected. Conjunctions subsumed by an adopted single literal are dropped. Adopted guards are stored as data: "opt": "start_cooking", "channel": "inferred", "when": "tgt_pot_state": "empty", "support": 31887 "opt": "start_cooking", "channel": "inferred", "when": "held": "dish", "support": 24240 "opt": "deliver", "channel": "explicit", "when": "__reason__": "NoPath", "support": 20884 Stage 4: menu assembly Fixed code interprets the guard file: the menu instantiates each option once per target of its class (never nearest-target only), evaluates every adopted conjunction against each instance’s read-set features, and hides matches; explicit-channel guards are re-checked live (e.g., NoPath by running the path planner). Finally, menu letters are shuffled per decision with an episode-seeded RNG: letter assignment is arbitrary, and a fixed construction order would leak option identity through position, which models exploit through a strong letter-A prior. Training rollouts expand options with the same raw executor the guards were mined against, so certified-valid temporally extended behaviors (e.g., walking to a still-cooking pot that finishes en route) remain executable. The outer contract-repair loop Stages 2–4 form the inner, fully mechanical loop. The outer loop exists because the contract itself—the Stage-1 schema—can be defective, and a defective contract corrupts the data the inner loop learns from. The division of labor mirrors the rest of the pipeline: the mechanical layers detect a defect (it surfaces as an impossible pattern in the burn-in log or the certificate), the frontier model repairs the schema, and Stages 2–3 rerun. Three defect families arose in development; we walk through each as bug, symptom, and fix. Defect 1: absolute-form effect predicates. The effect of pick_onion was first written in absolute form: after execution, held = onion. The bug: an agent that already holds an onion and executes pick_onion performs a silent no-op—yet the predicate is still true (it does hold an onion), so the vacuous execution is logged as a success. Symptom: these false successes flow into the certificate and veto the correct guard—“hide pick_onion when held = onion” is rejected for hiding logged “successes” that were never real. Fix: rewrite every acquire-type effect in delta form, pre ≠ X ∧ post = X. The no-op case now logs as effect_fail, and the missing guard is adopted on the next synthesis pass. Defect 2: shared-state attribution. The effect of place_in_pot was first “pot count increases.” In two-agent burn-in, while agent A is still walking to the pot, agent B places an onion; the pot count rises, and A’s in-flight option is logged as a success although A did nothing. Symptom: forged successes, again vetoing correct guards through the certificate. Fix: bind every effect to a private transition of the acting agent (my held item goes onion → None, and the pot count rose); options with no private component (start_cooking) are synthesized from solo-episode records only. Defect 3: nearest-target instantiation. Options were first instantiated only for their nearest target. In Asymmetric Advantages the serving station nearest to one agent lies in the unreachable half of the kitchen, so deliver always failed with NoPath and effectively vanished from the menu—even though a reachable station existed farther away. Symptom: an option whose explicit failure rate is 100%100\% while the task it names is plainly achievable. Fix: instantiate one menu entry per target of each bounded class (every pot, every station); guards then filter per instance. Worked example: one decision on the trained system. A decision reproduced verbatim from a logged Overcooked Asymmetric Advantages rollout of the trained Qwen3-4B checkpoint under the auto-guard interface. The system message is fixed; menus are instantiated per target and letters are shuffled per decision. --- SYSTEM MESSAGE --- You are an agent in a 2-player cooperative Overcooked-AI kitchen. Goal: place 3 onions in a pot, start the cook, plate the soup with a dish, deliver to the serving station. Each delivery scores +20 (shared with partner). The action menu only lists currently-feasible actions. Pick one letter. Output format (exactly two lines, nothing else): state: <empty | filling | cooking | ready_to_plate | delivering> choice: <single capital letter> --- USER MESSAGE --- You are player 1. World position: (1, 1). Facing: West. Holding: onion. Partner (player 0): pos=(5, 2) facing=North holding=onion. Pots: - pot@(4, 2): 0/3 onions you: 4 | partner: 1 (partner closer) - pot@(4, 3): 0/3 onions you: 5 | partner: 2 (partner closer) Counters: (empty) Layout: asymmetric_advantages. Step 4/150. Memory: - Last 5 macros decided: pick_onion - Total reward last 100 steps: 0 - Partner: pos=(5, 2) facing=North holding=onion Action menu: A) place held item in pot @ (4, 2) B) drop held item on counter @ (1, 0) C) place held item in pot @ (4, 3) D) drop held item on counter @ (2, 1) Choose one. Reply with exactly two lines. --- MODEL RESPONSE (verbatim) --- state: cooking choice: A Three interface properties are visible. The pot options (A, C) are instantiated once per target, and the counter-drop options (B, D) appear un-gated: the auto pipeline adopts no usability gate on hand-offs, since dropping while holding an item is a certified-valid execution. Letters carry no meaning across decisions. And the response contract is two plain lines; in the same logged episode, one of the eight recorded decisions violated the format and failed to parse. 5.2 Per-Agent LoRA and MAGRPO The collapse of §4 has two distinct components, and they call for different fixes. The first is a decoding-level prior attractor: the frozen base concentrates its output distribution on one “safe” action. On-policy policy gradient cannot repair this attractor—it sharpens it, because only sampled actions receive gradient. The option interface of §5.1 attacks this component by reshaping the action space. The second component arises in the multi-agent setting: with a single shared adapter, all agents remain copies of one policy—their rollouts explore only symmetric behavior, conflicting role gradients average into the same weights, and the team settles into a joint local optimum. This happens at the macro level even with the option interface in place. Our fix simply makes agents able to differ: each agent i owns a private LoRA adapter θi _i on the shared frozen base f, so πθi(zi∣oi)=f∘θiπ^i_ _i(z^i o^i)=f _i, and each adapter is updated only on its own agent’s decisions. Differences introduced by rollout stochasticity can then accumulate in private parameters rather than average away, letting the team escape the symmetric attractor when the task rewards it. We train these adapters with Multi-Agent Group Relative Policy Optimization (MAGRPO); because each agent carries a private LoRA adapter on a shared frozen base, we refer to this per-agent variant as PA-MAGRPO. At each training round we collect K parallel joint rollouts of length T under the current joint policy, using the option-menu interface from §5.1 as the action interface. Each rollout is scored by a shared team return RkR_k – the team’s task reward for that episode (total apples eaten in Cleanup and Commons Harvest, total soups delivered in Overcooked) – identical for all agents in that rollout, and the group-relative advantage is computed by normalizing RkR_k within the cohort. The word “per-agent” refers to the parameterization and update, not to a separate reward: each agent has its own adapter, and each surrogate loss is evaluated only on that agent’s decisions (ok,ti,zk,ti)\(o^i_k,t,z^i_k,t)\. Specialization can therefore emerge because private adapters experience different observation–action histories under the same team advantage, rather than because cleaners and eaters receive different hand-designed rewards. Updates occur at option boundaries only; with episodic, undiscounted team returns this needs none of the semi-MDP discounting machinery. The full per-round procedure is given in Algorithm 1. Algorithm 1 states one PA-MAGRPO training round in full. Each round collects K joint rollouts under the current joint policy over the option menu, scores every rollout by the shared team return, and normalizes those returns within the cohort to obtain a single group-relative advantage per rollout—the same value for every agent. Each agent then takes PPO-clipped epochs over its own option decisions under that shared advantage, with a KL penalty toward the reference policy, and only its private LoRA adapter is updated. Updates occur at option boundaries only; with episodic, undiscounted team returns this requires none of the semi-MDP discounting machinery. Algorithm 1 MAGRPO training round (per-agent LoRA over option-menu rollouts) 1: input: base LLM f, per-agent adapters θii=1N\ _i\_i=1^N, reference policy πref _ref, option set Z, rollouts K, episode length T, clip ϵε, KL weight β, PPO epochs E. 2: for k=1,…,Kk=1,…,K do 3: Reset env; for t=1,…,Tt=1,…,T, each agent i samples an option zk,ti∼πθi(⋅∣ok,ti)z^i_k,t π^i_ _i(· o^i_k,t) from the option menu i(ok,ti)Z^i(o^i_k,t), the symbolic planner expands it into environment primitives, and the env returns rk,tr_k,t. 4: end for 5: For each rollout k, accumulate the shared team return Rk=∑tR(sk,t,k,t)R_k= _tR(s_k,t,a_k,t). 6: Compute cohort statistics μ=1K∑kRk,σ=stdkRk,μ= 1K\! _kR_k,\;\;σ=std_k\,R_k, and the shared group-relative advantage A^k,t=Rk−μσ. A_k,t\;=\; R_k-μσ. 7: For each agent i, take E PPO-clipped epochs over its own decisions (ok,ti,zk,ti)\(o^i_k,t,z^i_k,t)\ to minimize ℒi(θi)=−[min(ρk,tiA^k,t,clip(ρk,ti,1−ϵ,1+ϵ)A^k,t)]+βKL(πθi∥πref),L^i( _i)=-E\! [ \! (ρ^i_k,t A_k,t,\\ clip(ρ^i_k,t,1-ε,1+ε)\, A_k,t ) ]+β\,KL\! (π^i_ _i\,\|\, _ref ), where ρk,ti=πθi(zk,ti∣ok,ti)/πθi,oldi(zk,ti∣ok,ti)ρ^i_k,t=π^i_ _i(z^i_k,t o^i_k,t)\,/\,π^i_ _i,old(z^i_k,t o^i_k,t) is the per-decision importance ratio, and update θi _i. 8: Save per-agent LoRA checkpoints 6 Experiments 6.1 Setup Table 1: Cross-base PA-MAGRPO robustness across three games. Values are mean ± std over five independent training seeds. For each base model, the first row is the frozen base baseline (all zeros) and the second row is our method (per-agent LoRA + option menu + MAGRPO). Base model Cleanup (apples) Overcooked A (deliveries) Harvest Qwen3-4B-Instruct-2507 0.00±0.000.00± 0.00 0.00±0.000.00± 0.00 0.00±0.000.00± 0.00 92.85±13.2392.85± 13.23 6.21±0.806.21± 0.80 85.41±6.6385.41± 6.63 Qwen3.5-2B 0.00±0.000.00± 0.00 0.00±0.000.00± 0.00 0.00±0.000.00± 0.00 105.69±50.71105.69± 50.71 7.31±1.147.31± 1.14 97.82±6.8297.82± 6.82 gemma-4-E4B-it 0.00±0.000.00± 0.00 0.00±0.000.00± 0.00 0.00±0.000.00± 0.00 60.25±39.4060.25± 39.40 7.08±1.157.08± 1.15 99.84±15.7699.84± 15.76 gemma-4-E2B-it 0.00±0.000.00± 0.00 0.00±0.000.00± 0.00 0.00±0.000.00± 0.00 68.62±41.5968.62± 41.59 6.60±0.846.60± 0.84 115.52±12.02115.52± 12.02 Table 2: Ablations on the fixed Qwen3-4B backbone. Values are mean ± std over 20 evaluation episodes; best scores are bold. Cleanup (apples) Overcooked A (deliveries) Harvest Frozen + native (Abl. C) 0.00±0.000.00± 0.00 0.00±0.000.00± 0.00 0.00±0.000.00± 0.00 Frozen + option menu (base) 0.00±0.000.00± 0.00 0.20±0.510.20± 0.51 76.63±18.2476.63± 18.24 Native + MAGRPO (Abl. A) 0.000.00† 0.000.00† 0.000.00† Compound-JSON + MAGRPO (Abl. D) 59.70±47.6659.70± 47.66 0.000.00† 91.74±16.4991.74± 16.49 Shared-LoRA + option menu (Abl. B) 0.10±0.450.10± 0.45 2.25±1.592.25± 1.59 90.44±18.1990.44± 18.19 Full method (ours) 93.25±16.8493.25± 16.84 6.80±1.896.80± 1.89 95.68±15.6395.68± 15.63 † Training collapsed at zero return for the entire training horizon; no positive checkpoint exists to evaluate. Environments. We evaluate on 3 cooperative spatial games. Melting Pot Cleanup (Leibo et al. 2017; Agapiou et al. 2022) (clean_up substrate, 7 agents, 150-step episodes): agents share a river that must be cleaned of dirt, and an orchard that grows apples once dirt is below a threshold. The team-versus-individual tension creates a classic social dilemma. Overcooked-AI (Carroll et al. 2019) (2 agents, 400-step episodes): we use two separated-kitchen layouts. In Asymmetric Advantages (A), the two cooks are separated but each can still reach all resources needed to complete soups; in Forced Coordination (FC), neither side can complete a soup alone, so agents must hand off onions across the counter. Commons Harvest (commons_harvest_no_thinning substrate, 7 agents, 400-step episodes): agents eat apples on 6 patches for reward, but they must conserve, or the apples never regenerate on the patches. Full details of each environment follow. Melting Pot Cleanup (clean_up substrate; 7 agents; 150-step episodes). Agents share a river that accumulates dirt and an orchard whose apples regrow only while river dirt stays below a threshold. Eating apples yields individual reward that contributes to the team return; cleaning yields none directly. The tension between harvesting now and cleaning for future growth makes the game a sequential social dilemma: sustained team reward requires some agents to forgo eating. Overcooked-AI (2 agents; 400-step episodes; two separated-kitchen layouts). Cooks assemble and deliver onion soups: fetch three onions into a pot, cook, plate, and deliver. In Asymmetric Advantages (A) the two cooks occupy separate kitchen halves but each half contains every resource needed to complete a soup alone; coordination can help but is not required. In Forced Coordination (FC) neither half is self-sufficient—one side has ingredient dispensers, the other the pots and serving window—so nothing can be delivered without counter hand-offs. A/FC thus form a within-game control pair in which the same recipe faces optional versus mandatory coordination. Commons Harvest (commons_harvest_no_thinning substrate; 7 agents; 400-step episodes). Agents eat apples growing on six shared patches; an apple regrows only while at least one apple remains on its patch, so exhausting a patch destroys it permanently. Team return favors restraint over the 400-step horizon, but each individual bite is immediately rewarded—a common-pool resource dilemma. Option-library sampling. Frontier-model outputs cannot be seeded, so the construction prompt (§5.1) is sampled repeatedly and each training seed draws one candidate library at random—our results are not contingent on any single hand-polished library. Models and implementation. Our main experiments train the recipe across four small backbones (Qwen3-4B-Instruct-2507, Qwen3.5-2B, gemma-4-E4B-it, gemma-4-E2B-it; “Qwen3-4B” hereafter always denotes the first) with five seeds each. The ablation suite then fixes that same Qwen3-4B backbone, so that comparisons isolate the effect of the action interface and fine-tuning design rather than differences between backbones. For Cleanup and Commons Harvest we use LoRA rank 32, learning rate 5×10−55\!×\!10^-5, K=4K\!=\!4 rollouts per round, and sampling temperature 0.70.7. For Overcooked-AI we use LoRA rank 16, learning rate 2×10−52\!×\!10^-5, K=6K\!=\!6, sampling temperature 0.90.9, and KL coefficient 0.050.05. All runs use PPO clip ϵ=0.2ε=0.2, two inner epochs, advantage normalization, and gradient clipping at 1.01.0. Training runs on a single H100. These values were chosen from small pilot runs for training stability under a fixed compute budget and then held fixed across all methods compared within the same environment; no per-method tuning was performed. Within a game, the full method and its ablations share base model, rollout budget, optimizer settings, temperature, checkpoint rule (highest training-time team return), and evaluation protocol; ablations differ only in the removed component. Final evaluation uses 20 episodes with seeds disjoint from training. In the multi-seed, multi-backbone study, a training seed controls option-library sampling, LoRA initialization, environment initial states, and minibatch order, so each seed is an independent end-to-end run. The ablation suite (Table 2) uses a single training seed on the fixed Qwen3-4B backbone to keep the six-cell design affordable; the cross-base study (Table 1) tests whether the pattern persists across seeds and model families. What the training seed controls. A training seed fixes every stochastic component of one end-to-end run, and its influence enters through four channels. (i) Interface: the seed selects one of the retained option-library designs, so different seeds train against different option sets, menu phrasings, and per-decision letter shuffles—a different effective action space. (i) Initialization: per-agent LoRA weights are initialized from the seed, which determines the initial symmetry-breaking among the N otherwise identical agents and hence which agent drifts toward which role. (i) Rollout stochasticity: environment initial states and the temperature-based sampling of options during the K rollouts are seeded; because MAGRPO advantages are group-relative, whichever behavior first scores above the cohort mean is amplified. (iv) Optimization order: minibatch shuffling within the PPO epochs. Channels (i)–(iv) make each seed an independent optimization trajectory even under a fixed interface; channel (i) additionally varies the interface itself. The per-seed standard deviations in the cross-base table of the main paper therefore reflect the variability of the entire pipeline, not evaluation noise alone; evaluation episodes use a fixed held-out seed set and are unaffected by the training seed. Evaluation protocol and statistics. Every cell of the ablation table in the main paper is a 20-episode evaluation on the fixed seed set 10,000,10,001,…,10,019\10,000,10,001,…,10,019\, disjoint from the training seed range, so all rows are comparable seed-for-seed. Headline cells satisfy 95% percentile bootstrap confidence intervals that exclude zero (ours, Cleanup: 93.25[86.05, 100.85]93.25\,[86.05,\,100.85]; ours, Overcooked A: 6.80[5.95, 7.60]6.80\,[5.95,\,7.60]; 10,00010,000 resamples each). All pairwise differences between the full method and the frozen baselines are significant at p<10−7p<10^-7 under a two-sided Mann–Whitney U test. Because the ablation suite uses a single training seed, those intervals quantify evaluation-side variability only; the cross-base study (Table 1) and both behavioral figures aggregate five independent training seeds. Ablations. The ablation suite was run on a slightly earlier interface version whose guards were hand-selected rather than synthesized; since the ablations isolate the value of the menu and of per-agent capacity—not how guards are obtained—we report them unchanged. We compare six cells drawn from the design space abstraction: option menu / compound-JSON / native actions × capacity: per-agent LoRA / single shared LoRA / no training. • Full method (ours) — option menu + per-agent LoRA + MAGRPO. • Ablation A (native + MAGRPO) — drop the abstraction, keep per-agent LoRA, train with MAGRPO over raw motor actions. • Ablation B (option menu + shared LoRA + MAGRPO) — keep the abstraction, replace seven adapters with one shared adapter; all agents back-propagate into the same LoRA. • Ablation C (frozen, native) — no training, no abstraction; sanity check that the base model alone cannot play. • Ablation D (compound-JSON + per-agent LoRA + MAGRPO) replaces the menu with free structured generation over the same family of compound actions: the LLM emits a JSON object specifying an action type and arguments, executed when valid. This keeps macro-level action semantics but removes feasibility filtering and discrete menu selection—separating the value of macro actions from the value of the state-conditioned menu. 6.2 Results Main result: the recipe works across backbones and seeds. Across four small backbones × five independent training seeds (Table 1), the frozen base collects zero in every row of every game, while our full method obtains positive return for every backbone: Cleanup 60.2560.25–105.69105.69, Overcooked Asymmetric Advantages 6.216.21–7.317.31, Commons Harvest 85.4185.41–115.52115.52. Gain magnitudes vary with backbone and seed—interface design and initialization still affect optimization—but the qualitative pattern is stable: the frozen base never plays; menu-based per-agent fine-tuning consistently produces non-trivial behavior. Headline numbers on the fixed backbone. The full method delivers large gains over the frozen base on every game we evaluate (Table 2). On Melting Pot Cleanup the team rises from 0.00±0.000.00\!±\!0.00 apples (frozen base with raw motor actions) to 93.25±16.8493.25\!±\!16.84 apples per 150-step episode. On Overcooked-AI Asymmetric Advantages deliveries rise from 0.20±0.510.20\!±\!0.51 to 6.80±1.896.80\!±\!1.89 per 400-step episode – a 34×34× improvement. On Commons Harvest the team reaches 95.68±15.6395.68\!±\!15.63 apples. Every “ours” cell is statistically separated from its frozen and zero-shot cells. Removing the option menu is catastrophic (Ablation A). Training the same per-agent LoRAs with MAGRPO directly on raw motor actions, with the option menu switched off, collapses every cell to zero on all three games and never produces a positive checkpoint. This confirms the statement of §5.2: at the raw-action timescale the 4B base rarely emits a coherent multi-step trajectory under random exploration, so cohort returns are nearly always zero, the GRPO group standard deviation collapses, and updates carry no useful signal—gradients cannot repair what sampling never reaches. A single shared LoRA cannot hold role specialization (Ablation B). Keeping the menu but sharing one adapter recovers the reward signal, yet the joint policy never stabilizes: Cleanup averages 0.10±0.450.10\!±\!0.45 apples at its best checkpoint (isolated rounds spike to ∼55 55, then revert—the adapter can briefly express one role but cannot hold cleaner and eater at once), and Overcooked reaches 2.25±1.592.25\!±\!1.59 deliveries, about a third of the full method. The symmetric Commons Harvest is forgiving (90.44±18.1990.44\!±\!18.19), so the shared-vs.-per-agent gap is specifically about role differentiation, not capacity—confirming the statement of §5.2 that a shared adapter re-creates the collapse at the macro level even with the option interface in place. The frozen base cannot play (Ablation C, and the “Frozen + option menu” baseline). With no training and no abstraction the 4B base scores zero on every game, as expected. Adding the option menu to the frozen base (no MAGRPO) lifts Harvest to 76.63±18.2476.63\!±\!18.24 but leaves Cleanup at 0.000.00 and Overcooked at 0.20±0.510.20\!±\!0.51. This isolates the contributions cleanly: the abstraction alone is sufficient to play the easy, symmetric substrate (Harvest), but the harder cooperative games (Cleanup, Overcooked) additionally require MAGRPO fine-tuning of per-agent LoRAs. Compound actions without the menu filter are unstable (Ablation D). Free-form compound-JSON generation—same macro semantics, no feasibility filter—partially trains on Cleanup (59.70±47.6659.70\!±\!47.66, vs. ours 93.25±16.8493.25\!±\!16.84; the standard deviation is nearly as large as the mean) but never escapes zero on Overcooked; symmetric Harvest is again forgiving (91.74±16.4991.74\!±\!16.49). Without the filter the model frequently emits compound actions that are illegal or no-ops in the current state (§6.3). Across the four ablations, neither ingredient is redundant, as claimed in the intro: the menu is what produces any non-zero return on the harder games, and per-agent LoRA is what lets the cohort hold a heterogeneous joint policy once it exists. Harvest needs neither roles nor workflows to collect an apple, so it performs decently even untuned; what matters there is conservation, examined in §6.4. 6.3 Trajectory-level analysis: why and how it works To understand the gap between our method and the failed ablations, we record full per-step trajectories. (1) The abstraction collapses the effective action space. In the native-action cells (A, C) every environment step triggers an LLM call (call ratio 1.001.00). In the option-menu cells (B, ours) the ratio drops to ∼0.40 0.40 on Cleanup and Commons Harvest and 0.710.71 on Overcooked: each LLM commitment spans 22–33 env steps, so the MAGRPO advantage propagates over a 22–3×3× shorter decision sequence. (2) Native-action MAGRPO collapses to a single letter. The frozen base and native-action MAGRPO cells spend nearly all decisions on one motor action (FIRE_CLEAN: 95.8%95.8\% frozen, 98.4%98.4\% after MAGRPO on Cleanup; EAST: 58.8%58.8\%/60.2%60.2\% on Overcooked). RL over raw letters does not learn a better policy—it sharpens an already-degenerate mode, exactly as claimed in §5.2: only sampled actions receive gradient, so on-policy RL reinforces the prior attractor instead of repairing it. (3) Shared LoRA trains, but symmetry caps it. The shared-LoRA cell does escape zero reward, but a single adapter cannot hold two roles at once: every agent converges to the same macro distribution. On Cleanup its training curve shows brief reward spikes when the one policy happens to express a useful role (everyone cleans, or everyone eats), followed by reversion as the complementary work goes undone—a symmetric policy that looks locally sensible per agent yet makes an inefficient team, exactly the mechanism per-agent adapters remove. Figure 1: Per-agent macro choices on Overcooked A. Agent 0 concentrates on onion loading (≈69%≈ 69\%); agent 1 waits ≈72%≈ 72\% of the time; counter operations are zero for both—no hand-off, a one-worker/one-idler split. Figure 2: Commons Harvest patch survival. The frozen base depletes all patches by ∼ step 200; the tuned checkpoint keeps ∼2.5 2.5 of 66 alive through step 400—restraint despite a nearly flat reward. Figure 3: Per-agent visit heatmaps for Cleanup (ours). With no role labels, a1,a3,a6a_1,a_3,a_6 specialize as eaters (orchard rows) and a0,a4,a5a_0,a_4,a_5 as cleaners (river shore); a2a_2 is near-stationary. 6.4 Emergent behavior: reward and cooperation decouple Beyond why the method trains, the trajectories reveal what the agents do: under one fixed recipe, genuine role specialization on Cleanup, no coordination on Overcooked, and resource restraint on Commons Harvest. Cleanup: emergent role specialization. The seven Cleanup adapters of our method partition cleanly into “cleaner” and “eater” roles without any role label: per-agent apple counts split into three eaters (∼57 57–6060 apples each), three cleaners (all 0), and one near-stationary helper, and the visit heatmaps (Fig. 3) confirm the spatial split—eaters on the orchard rows, cleaners on the river shore. Shared LoRA (Abl. B) cannot reproduce this: its per-agent breakdown is essentially uniform. Strikingly, compound-JSON with per-agent LoRA (Abl. D) also fails to specialize—all seven adapters eat apples, and the identity of the dominant eater shifts from episode to episode. Per-agent capacity is therefore necessary but not sufficient: the constrained pick of the option menu is what lets each adapter commit to a distinct typed-option distribution that gradient descent can sharpen agent-by-agent, whereas free-form (x,y)(x,y) generation diffuses the gradient and the partition collapses. Overcooked: high reward without coordination. On Overcooked-AI Asymmetric Advantages (Table 2), each agent can reach every resource, and the trained team reaches high reward without coordinating. The per-agent macro histogram (Fig. 1) shows two signatures: no item hand-off (counter operations are 0%0\% for both agents), and a lopsided workload—agent 0 spends ∼69% 69\% of choices loading onions while covering every downstream stage, whereas agent 1 waits on 72%72\% of its choices. Run alone in the same layout, both agents still deliver (44 and 22), so each has independently learned the full task: Asymmetric Advantages is solo-solvable, and reward here overstates cooperation. Forced Coordination removes the solo solution: neither cook can complete a soup alone. There the frozen base scores 0.000.00 while the trained checkpoint reaches 7.57±1.167.57± 1.16 deliveries, and the same solo test flips—Forced-Coordination agents placed alone deliver 0 each. Their positive team deliveries are behavior that only functions with a partner—direct evidence of emerged coordination. Commons Harvest: restraint without reward. Because apple count does not reveal whether the shared patches survive, we track patch survival in a separate 400-step evaluation. Apple reward moves modestly (frozen 76.73±16.2976.73± 16.29 vs. tuned 98.29±14.5298.29± 14.52), but patch survival changes sharply (Fig. 2): the frozen base depletes all six patches by roughly step 200200, while the tuned checkpoint keeps about 2.52.5 alive through step 400400. The mechanism is restraint: roughly two agents harvest while the rest wait, trading immediate consumption for persistence the reward curve barely registers. 7 Conclusion and Limitations Casting small (2–4B) LLMs as policies over a frontier-drafted library of symbolic options and training a private LoRA adapter per agent with PA-MAGRPO makes them trainable as cooperative spatial policies (Cleanup apples 0.00→92.850.00\!→\!92.85; Overcooked deliveries 0.00→6.210.00\!→\!6.21). Used as a fixed instrument, the same recipe reveals the decoupling: high reward coincides with real specialization on Cleanup but with a one-worker/one-idler team on Overcooked, nearly flat reward hides real restraint on Commons Harvest, and coordination appears when—and only when—the layout requires it. Team reward alone cannot certify cooperation; behavioral metrics must sit alongside it. Limitations. Our results cover small (≤ 4B) LLMs and three spatial games; transfer to larger scales or non-spatial tasks is untested, the option library is per-game, and we lack a principled way to choose an option space. The decoupling finding is behavioral evidence, not a general theory: the cross-game pattern replicates over five seeds and four backbones, but the specific role assignment is checked on one seed, and the Forced-Coordination control assumes a non-solo-solvable layout. Generative AI Disclosure Beyond the frontier model used inside the method itself (Claude Opus 4.8; §5.1), large language models (Claude Opus 5 and Claude Fable 5, Anthropic) were used to edit and revise the manuscript draft. The authors remain fully responsible for all content, and all references have been verified against their original sources. References Agapiou et al. (2022) Agapiou, J. P.; Vezhnevets, A. S.; Duéñez-Guzmán, E. A.; Matyas, J.; Mao, Y.; Sunehag, P.; Köster, R.; et al. 2022. Melting Pot 2.0. arXiv preprint arXiv:2211.13746. Ahn et al. (2022) Ahn, M.; Brohan, A.; Brown, N.; et al. 2022. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. In Conference on Robot Learning (CoRL). Amato, Konidaris, and Kaelbling (2014) Amato, C.; Konidaris, G.; and Kaelbling, L. P. 2014. Planning with Macro-Actions in Decentralized POMDPs. In Proceedings of the 13th International Conference on Autonomous Agents and Multiagent Systems (AAMAS). Biswas et al. (2026) Biswas, U.; Palod, V.; Bhambri, S.; and Kambhampati, S. 2026. Who Is Helping Whom? Analyzing Inter-Dependencies to Evaluate Cooperation in Human-AI Teaming. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 17347–17356. Carroll et al. (2019) Carroll, M.; Shah, R.; Ho, M. K.; Griffiths, T. L.; Seshia, S. A.; Abbeel, P.; and Dragan, A. 2019. On the Utility of Learning about Humans for Human-AI Coordination. In Advances in Neural Information Processing Systems (NeurIPS). Gallego (2026) Gallego, V. 2026. Cooperation and Exploitation in LLM Policy Synthesis for Sequential Social Dilemmas. arXiv preprint arXiv:2603.19453. Hu et al. (2022) Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR). Hua et al. (2025) Hua, Y.; Chen, H.; Wang, S.; Li, W.; Wang, X.; and Luo, J. 2025. Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agents. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:2506.07388. Huang et al. (2022) Huang, W.; Xia, F.; Xiao, T.; et al. 2022. Inner Monologue: Embodied Reasoning through Planning with Language Models. In Conference on Robot Learning (CoRL). Hughes et al. (2018) Hughes, E.; Leibo, J. Z.; Phillips, M.; Tuyls, K.; Duéñez-Guzmán, E. A.; Castañeda, A. G.; Dunning, I.; Zhu, T.; McKee, K. R.; Koster, R.; et al. 2018. Inequity Aversion Improves Cooperation in Intertemporal Social Dilemmas. In Advances in Neural Information Processing Systems (NeurIPS). Lee, Cho, and Choi (2026) Lee, W.; Cho, J.; and Choi, J. 2026. MapCoder-Lite: Distilling Multi-Agent Coding into a Single Small LLM. In Findings of the Association for Computational Linguistics: EACL. ArXiv:2509.17489. Leibo et al. (2017) Leibo, J. Z.; Zambaldi, V.; Lanctot, M.; Marecki, J.; and Graepel, T. 2017. Multi-Agent Reinforcement Learning in Sequential Social Dilemmas. In Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AAMAS). Liang et al. (2023) Liang, J.; Huang, W.; Xia, F.; Xu, P.; Hausman, K.; Ichter, B.; Florence, P.; and Zeng, A. 2023. Code as Policies: Language Model Programs for Embodied Control. In IEEE International Conference on Robotics and Automation (ICRA). Liu et al. (2026) Liu, S.; Liang, Z.; Lyu, X.; and Amato, C. 2026. LLM Collaboration with Multi-Agent Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence. ArXiv:2508.04652. Ma et al. (2024) Ma, H.; Hu, T.; Pu, Z.; Liu, B.; Ai, X.; Liang, Y.; and Chen, M. 2024. Coevolving with the other you: Fine-tuning llm with sequential cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 37: 15497–15525. Mosquera et al. (2025) Mosquera, M.; Pinzon, J. S.; Rios, M.; Fonseca, Y.; Giraldo, L. F.; Quijano, N.; and Manrique, R. 2025. Can LLM-Augmented Autonomous Agents Cooperate? An Evaluation of Their Cooperative Capabilities through Melting Pot. IEEE Transactions on Artificial Intelligence. Pérolat et al. (2017) Pérolat, J.; Leibo, J. Z.; Zambaldi, V.; Beattie, C.; Tuyls, K.; and Graepel, T. 2017. A Multi-Agent Reinforcement Learning Model of Common-Pool Resource Appropriation. In Advances in Neural Information Processing Systems (NeurIPS). Piche et al. (2025) Piche, D.; Muqeeth, M.; Aghajohari, M.; Duque, J.; Noukhovitch, M.; and Courville, A. 2025. Learning Robust Social Strategies with Large Language Models. arXiv preprint arXiv:2511.19405. Shao et al. (2024) Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Sutton, Precup, and Singh (1999) Sutton, R. S.; Precup, D.; and Singh, S. 1999. Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artificial Intelligence, 112(1-2): 181–211. Valmeekam et al. (2023) Valmeekam, K.; Marquez, M.; Sreedharan, S.; and Kambhampati, S. 2023. On the Planning Abilities of Large Language Models – A Critical Investigation. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:2305.15771. Xiao, Hoffman, and Amato (2020) Xiao, Y.; Hoffman, J.; and Amato, C. 2020. Macro-Action-Based Deep Multi-Agent Reinforcement Learning. In Proceedings of the 3rd Conference on Robot Learning (CoRL), volume 100 of Proceedings of Machine Learning Research, 1146–1161. Zhang, Kapoor, and Sun (2025) Zhang, B.; Kapoor, A.; and Sun, M. 2025. Low-Rank Agent-Specific Adaptation (LoRASA) for Multi-Agent Policy Learning. arXiv preprint arXiv:2502.05573. Zhuang et al. (2025) Zhuang, Y.; Shen, Y.; Zhang, Z.; Chen, Y.; and Miao, F. 2025. YOLO-MARL: You Only LLM Once for Multi-agent Reinforcement Learning. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

Instant research discovery