Paper deep dive
MATES: Learning Multi-Agent Interactions by Transforming Observations for Frozen Single-Agent Policies
Elie Abboud, Oren Gal
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/23/2026, 3:39:42 AM
Summary
The paper introduces MATES (Multi-Agent Observation Transformation for Existing Single-Agent Policies), a framework that adapts frozen single-agent policies for multi-agent reinforcement learning (MARL) by learning a lightweight observation adapter. This approach allows agents to leverage pre-existing individual competence without retraining the core policy, optimizing only 3.5-7.3% of parameters while outperforming training from scratch and matching full fine-tuning performance across various tasks and team sizes.
Entities (11)
Relation Signals (9)
MATES → freezes → Solo Policy
confidence 95% · The policy π solo is trained exclusively in the single-agent setting and remains frozen upon transfer to the multi-agent setting.
MATES → uses → Observation Adapter
confidence 95% · MATES places a trainable observation adapter C φ before the frozen policy.
MATES → evaluatedon → VMAS Navigation
confidence 92% · We evaluate MATes... on... navigation... VMAS Navigation task...
MATES → evaluatedon → POGEMA
confidence 92% · We evaluate MATES using both on- and off-policy algorithms on lifelong pathfinding... POGEMA.
MATES → evaluatedon → VMAS Discovery
confidence 92% · We evaluate MATes... on... cooperative discovery... VMAS Discovery...
MATES → optimizes → 3.5-7.3% parameters
confidence 90% · MATES optimizes only 3.5-7.3% as many parameters as full-policy training...
MATES → outperforms → MARL training from scratch
confidence 90% · MATES ... consistently outperforming MARL training from scratch.
MATES → comparedwith → PegMARL
confidence 85% · remains competitive overall with PegMARL and R2BC.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-agent reinforcement learning (MARL) commonly trains decentralized policies from scratch, requiring agents to acquire individual task competence and coordination simultaneously. Yet many multi-agent problems admit a compatible single-agent counterpart in which the underlying task can be learned in isolation. We introduce Multi-Agent Observation Transformation for Existing Single-Agent Policies (MATES), an input-side adaptation framework for tasks whose multi-agent observations preserve the solo-task information while exposing separately identifiable neighbor information. From multi-agent experience, MATES learns a small adapter that maps this observation into the format expected by a frozen single-agent policy, inducing actions suited to the shared environment without updating the single-agent policy itself. MATES leaves the pretrained policy's internal architecture unchanged and retains the objectives and update procedures of the underlying MARL algorithm. We evaluate MATES using both on- and off-policy algorithms on lifelong pathfinding, navigation, and cooperative discovery, spanning discrete and continuous observation and action spaces. Across all evaluated settings, MATES optimizes only 3.5-7.3% as many parameters as full-policy training while consistently outperforming MARL training from scratch. It approaches the performance of full fine-tuning, remains competitive overall with demonstration-based baselines, and retains strong task performance at team sizes not encountered during training. These results provide evidence that, under this observation structure, effective multi-agent behavior can be learned without modifying the policy that encodes individual competence.
Tags
Links
- Source: https://arxiv.org/abs/2609.26010v1
- Canonical: https://arxiv.org/abs/2609.26010v1
Trouble viewing inline? Open PDF directly →
Full Text
37,787 characters extracted from source content.
Expand or collapse full text
MATES: Learning Multi-Agent Interactions by Transforming Observations for Frozen Single-Agent Policies Elie Abboud 1 and Oren Gal 2 Abstract— Multi-agent reinforcement learning (MARL) com- monly trains decentralized policies from scratch, requiring agents to acquire individual task competence and coordina- tion simultaneously. Yet many multi-agent problems admit a compatible single-agent counterpart in which the underlying task can be learned in isolation. We introduce Multi-Agent Observation Transformation for Existing Single-Agent Policies (MATES), an input-side adaptation framework for tasks whose multi-agent observations preserve the solo-task information while exposing separately identifiable neighbor information. From multi-agent experience, MATES learns a small adapter that maps this observation into the format expected by a frozen single-agent policy, inducing actions suited to the shared environment without updating the single-agent policy itself. MATES leaves the pretrained policy’s internal architecture unchanged and retains the objectives and update procedures of the underlying MARL algorithm. We evaluate MATES using both on- and off-policy algorithms on lifelong pathfinding, navigation, and cooperative discovery, spanning discrete and continuous observation and action spaces. Across all evaluated settings, MATES optimizes only 3.5–7.3% as many parameters as full-policy training while consistently outperforming MARL training from scratch. It approaches the performance of full fine-tuning, remains competitive overall with demonstration- based baselines, and retains strong task performance at team sizes not encountered during training. These results provide evidence that, under this observation structure, effective multi- agent behavior can be learned without modifying the policy that encodes individual competence. I. INTRODUCTION Deploying several capable agents together introduces a challenge that is not present when each acts alone: the actions of one agent change the conditions under which the others must succeed. During learning, this interdependence gives rise to non-stationarity, a central challenge in MARL [1]: as agents update their policies simultaneously in a shared environment, each must continually adapt to the changing behavior of the others. For example, in robot navigation (e.g. [2], [3]), a single agent may need only to navigate along a route toward its destination, whereas successful multi-agent navigation may additionally require avoiding collisions and coordinating with other agents [2], [4]. A common approach to these problems is to train decen- tralized policies from scratch directly in the multi-agent envi- ronment, requiring them to acquire task-specific competence and coordination simultaneously [1]. For instance, Indepen- dent Proximal Policy Optimization [5] and Independent Soft Actor-Critic (ISAC; [6], [7]) train agents using experience 1 Elie Abboud is with the department of Marine Technologies, University of Haifa, Israel. Email: eliabboud1000@gmail.com. 2 Oren Gal is with the Department of Marine Technologies, University of Haifa, Israel. Email: orengal@univ.haifa.ac.il. generated while all agents interact simultaneously with the shared environment. Many problems in the multi-agent setting have natural single-agent counterparts, in which the underlying individual task may be easier to learn. Nevertheless, standard MARL approaches that train decentralized policies from scratch do not attempt to utilize competence already acquired by an individual agent. For example, an agent trained to navigate a map by itself may already have learned to move toward its goal, follow a route, or search for a target. Training from scratch discards this existing capability and relearns both individual competence and coordination in the multi-agent environment. Although the effective dynamics and available observations change when other agents are introduced, we hypothesize that competence learned through single-agent interaction remains valuable. What changes is the context in which the resulting behavior must be expressed. This motivates our central research question: Research Question 1: Can effective multi-agent behavior be learned without modifying a policy that already encodes individual competence? We investigate this question in tasks whose observation structure reflects the distinction between individual com- petence and multi-agent coordination. Let o i denote the observation available to agent i in the single-agent setting. When the agent is deployed with teammates, its observation expands to include local information N i about neighboring agents, yielding (o i ,N i ). Thus, o i contains the information on which the agent originally learned its individual task competence, while N i provides the additional context needed to respond appropriately to teammates. This decomposition provides a natural interface for adapting the agent’s behavior to the multi-agent setting while preserving the policy that encodes its individual competence. To exploit this interface, we introduce Multi-Agent Ob- servation Transformation for Existing Single-Agent Policies (MATES), an input-side adaptation framework. A small adapter receives (o i ,N i ) and produces a transformed ob- servation ̃o i in the format expected by the frozen single- agent policy, which we denote by π solo . The policy π solo is trained exclusively in the single-agent setting and remains frozen upon transfer to the multi-agent setting. MATES uses multi-agent experience to optimize its lightweight trainable components while leaving the action-producing single-agent policy unchanged. It retains the objectives and update proce- dures of the underlying MARL algorithm, augmenting only the architecture with lightweight components and restrict- ing optimization to the designated trainable components. arXiv:2609.26010v1 [cs.MA] 22 Sep 2026 MATES thereby learns how the newly available neighbor information should transform the observation presented to π solo . Because this adaptation does not require modifying the internal architecture of π solo , MATES is compatible with a broad range of policy architectures. At execution time, each agent independently applies the shared adapter–policy composition to its own local observation, without relying on a centralized controller. Our contributions are as follows: (i) We introduce MATES, a general framework for trans- ferring a competent single-agent policy to multi-agent settings through lightweight observation adaptation. For tasks with a compatible single-agent counterpart, MATES can be integrated with MARL algorithms without changing their learning objectives or the in- ternal architecture of the pretrained policy. (i) We evaluate MATES on three tasks—lifelong pathfind- ing, navigation, and cooperative discovery—spanning discrete and continuous observation and action spaces. We instantiate its lightweight adapter as either a CNN or an MLP according to the observation modality and integrate it with both on-policy and off-policy MARL algorithms. Across all evaluated settings, MATES opti- mizes only 3.5–7.3% as many parameters as full-policy training while consistently outperforming MARL train- ing from scratch. (i) We show that MATES approaches the performance of full fine-tuning while updating substantially fewer parameters and remains competitive overall with PegMARL and R2BC. Whereas PegMARL and R2BC transfer prior behavior through demonstrations, MATES transfers competence by directly retaining the pretrained single-agent policy as a frozen controller, without requiring a demonstration buffer or imitation- learning updates. (iv) We demonstrate that MATES generalizes beyond its training team size, retaining strong task performance across agent counts not encountered during training. I. RELATED WORK A. Transfer and Demonstration-Guided Multi-Agent Learn- ing Transfer learning in reinforcement learning seeks to im- prove learning in a target task by reusing knowledge acquired from related source tasks. This knowledge can be transferred through policy reuse, policy distillation, value functions, ex- perience, models, and other learned representations; broader treatments of transfer in RL and MARL are provided by [8], [9]. Imitation learning (IL) seeks to reproduce an expert pol- icy from demonstrations generated by an expert. Behavior cloning (BC) is a commonly employed IL technique that estimates the expert policy through supervised learning on expert demonstrations [10], [11]. Round-Robin Behavior Cloning (R2BC) [12] extends BC to multi-agent systems by allowing an expert to control one agent at a time while the remaining agents execute their current policies. The resulting per-agent demonstrations are accumulated and used to update the policies through BC. Beyond direct imitation, demonstrations can also guide re- inforcement learning. In MARL, Personalized Expert-Guided MARL (PegMARL) [13] uses personalized single-agent demonstrations to train behavior and transition discriminators whose outputs provide reward-shaping signals for learning a multi-agent policy from scratch. R2BC and PegMARL therefore rely on demonstration data and imitation or demonstration derived learning signals. MATES uses neither: it preserves the pretrained single-agent policy itself as a frozen, action-producing controller and learns its observation-side adaptation directly from multi- agent experience. B. Adaptation of Frozen Models and Policies Parameter-efficient adaptation repurposes pretrained mod- els by optimizing a restricted set of parameters rather than fine-tuning the complete model. Common approaches in- clude adapter modules, low-rank updates, and prompt-based methods [14], [15]. Adapter tuning inserts trainable mod- ules within a frozen network, whereas side-tuning combines the output of a lightweight side network with that of the unchanged pretrained network [16], [17]. MATES instead leaves the internal computation of the pretrained policy unchanged and adapts the observation presented to it. Related policy-adaptation methods preserve an existing policy or controller while learning an additional component. Residual reinforcement learning adds a learned correction to the action of a fixed controller [18], [19]; MATES instead adapts the observation presented to the controller without directly correcting its actions. VR-Goggles for Robots trans- forms real-world images into the simulated visual domain expected by a pretrained policy [20]; MATES likewise oper- ates at the policy’s input, but seeks to adapt its behavior to multi-agent interaction rather than retain its learned behavior across a domain shift. I. PRELIMINARIES A. Decentralized Cooperative Multi-Agent Reinforcement Learning We model the task as a partially observable Markov game with n homogeneous agents: G = (S,A n , ̄ O n ,P, Ω,r i n i=1 ,ρ 0 ,γ,n). At timestep t, state s t ∈ S yields each agent a local observation ̄o i,t ∼ Ω i (· | s t ). Each agent i samples a i,t ∼ π θ (· | ̄o i,t ), forming the joint actiona t = (a 1,t ,...,a n,t ). The environment draws s t+1 ∼ P (· | s t ,a t ) and assigns r i (s t ,a t ,s t+1 ). We consider the setting in which agents share parameters. The objective is J (π θ ) =E τ∼π θ " T−1 X t=0 γ t n X i=1 r i (s t ,a t ,s t+1 ) # . Under decentralized training and decentralized execution (DTDE) we consider, neither the actors nor the critics receive the global state or joint action. B. Single-Agent Counterpart and Frozen Policy We consider tasks with a compatible single-agent coun- terpart sharing action space A. Its observation o i ∈ O solo retains the same task semantics after deployment, while the multi-agent observation is ̄o i = (o i ,N i ). Here, N i locally summarizes neighboring agents and may have fixed size rather than enumerate the team. A solo policy π solo (a i | o i ) is trained in this counterpart. During transfer, it is frozen: its parameters are not updated by multi-agent training, although gradients may pass through it to optimize components pre- ceding its input. We focus on problems admitting the explicit decomposi- tion ̄o i = (o i ,N i ) to maintain a clear separation between task-specific information and multi-agent interactions. This allows our experiments to directly evaluate whether MATES maps the multi-agent extension of the problem into the competency space of the frozen solo policy. The architecture itself does not require this decomposition and could, in principle, be applied whenever a compatible single-agent counterpart exists, although we do not evaluate that broader setting. IV. METHOD We now present MATES. As illustrated in Figure 1, the method proceeds in two sequential phases. First, a policy π solo,θ is trained in the single-agent counterpart to acquire task-specific competence. MATES then transfers this policy unchanged to the multi-agent task and freezes its parameters θ. Because multi-agent observation additionally provides information about neighboring agents that π solo,θ was not trained to process, MATES places a trainable observation adapter C φ before the frozen policy. For each agent i, the adapter maps the full multi-agent observation ̄o i = (o i ,N i ) to a solo-compatible observation ̃o i = C φ (o i ,N i ), ̃o i ∈O solo .(1) The resulting MATES policy is the composition π MATES,φ (a i | ̄o i ) = π solo,θ a i | C φ ( ̄o i ) .(2) Thus, neighboring-agent information affects action selec- tion by changing the effective observation of the solo policy rather than by modifying or replacing its learned controller. During the multi-agent phase, φ is optimized using the objective of the underlying MARL algorithm. Although π solo,θ is frozen, it remains part of the differen- tiable computation graph. Gradients therefore pass through the policy with respect to its input and update C φ , while no gradient updates are applied to θ. Consequently, multi- agent experience teaches the adapter which transformations of (o i ,N i ) elicit effective behavior from the preserved single- agent controller. See Algorithm 1, describing MATES in psuedocode. Algorithm 1 Multi-Agent Observation Transformation for Existing Single-Agent Policies (MATES) Require: Compatible single-agent task G solo and multi- agent task G MA 1: π solo,θ ← RL-TRAIN(G solo ) 2: Freeze θ and initialize the observation adapter C φ 3: Define π MATES,φ (a i | ̄o i ) = π solo,θ a i | C φ ( ̄o i ) 4: φ ⋆ ← MARL-TRAIN(G MA ,π MATES,φ ;θ frozen) 5: return π MATES,φ ⋆ A. Implementation Variants MATES does not require the observation adapter to follow a particular parameter-sharing scheme. A shared adapter C φ may be applied to every agent, as in Algorithm 1, or each agent may have an independently parameterized adapter C φ i . Sharing is natural for homogeneous agents and keeps the number of trainable parameters independent of the team size, whereas independent adapters permit agent- specific transformations. We employ shared adapters in our experiments. Actor–critic methods additionally require the critic to operate under the multi-agent observation distribution. The critic learned during single-agent training is valuable for the same reason as the solo policy: it has already learned to evaluate progress in the underlying task. We therefore transfer and freeze the solo critic and place a separately initialized observation adapter before it. For a value-function critic, the actor and critic become π MATES (a i | ̄o i ) = π solo,θ π a i | C π φ π ( ̄o i ) ,(3.1) V MATES ( ̄o i ) = V solo,θ V C V φ V ( ̄o i ) ,(3.2) where C π φ π and C V φ V are distinct adapters and θ π and θ V remain frozen. For an action-value critic, the corresponding construction is Q MATES ( ̄o i ,a i ) = Q solo,θ Q C Q φ Q ( ̄o i ,a i ).(4) The underlying actor and critic objectives optimize their respective adapters. When multiple critics are used, each critic receives its own independently initialized adapter. We also consider optionally training the transferred critic’s output head while keeping the remainder of the critic frozen. This allows the final value prediction to be recalibrated for the multi-agent task without relearning the critic’s internal representation. We refer to the variant that trains this head as MATES and to the variant that keeps it frozen as MATES- NH 1 . As we will see in our experiments, this additional flexibility has no interpretible effect on performance. V. EXPERIMENTS A. Research Questions We organize our evaluation around four research ques- tions: 1 MATES-NH expands to MATES-No Head trained. 1. Single-agent RL Learn individual task competence Single-agent observation o i π solo θ: TRAINABLE Action a i TRANSFER FREEZEθ 2. Multi-agent RL Learn a teammate-aware observation transformation Full multi-agent observation (o i , N i ) C φ φ: TRAINABLE π solo θ: FROZEN Action a i ̃o i optimized in this phaseparameters fixed Fig. 1: Overview of MATES. Training proceeds in two sequential phases. First, the parameters θ of a single-agent policy π solo are trained normally using the single-agent observation o i . Second, π solo is transferred to the multi-agent setting and θ is frozen, while the parameters φ of an observation adapter C φ are trained using the underlying MARL method to map the full multi-agent observation ̄o i = (o i ,N i ) to a transformed observation ̃o i consumed by π solo . Solid green and dashed gray boxes denote trainable and frozen components, respectively. The figure depicts the action-selection pathway; critic-side components are omitted for clarity. RQ1 Performance and parameter efficiency: Can MATES attain competitive multi-agent task performance while optimizing only a small fraction of the parameters of a conventional MARL policy during transfer? RQ2 Generalization across team sizes: Does an adapter trained at one team size remain effective when deployed with different, including substantially larger, numbers of agents? RQ3 Source of performance: Does MATES benefit specifi- cally from transferring a competent solo policy, or can comparable performance be obtained from the compact adapter architecture alone? RQ4 Value-head adaptation: Is training the transferred critic’s value head necessary, or is adapting its observa- tion sufficient? B. Tasks and Metrics We evaluate MATES on three partially observable multi- agent tasks spanning one discrete grid navigation environ- ment and two continuous multi-robot control. POGEMA. [2] is a grid-based multi-agent pathfinding environment in which agents navigate toward assigned goals while avoiding obstacles and one another. Upon reaching a goal, an agent is assigned another, requiring policies to sustain navigation over time. We measure task completion using throughput, defined as the number of goals reached per environment step; higher values indicate better performance. Navigation. [3] The VMAS Navigation task places continuous-control agents at randomly sampled positions and assigns each agent a goal. Agents must reach their respective goals while avoiding collisions, using local state and range- sensor observations. When all agents simultaneously reach their goals, the episode ends. We measure task success as the number of steps required to complete an episode; lower values indicate faster task completion. Discovery. [3] In VMAS Discovery, agents must locate and cover targets while avoiding collisions. Covered targets are respawned, requiring agents to continually redistribute themselves throughout the environment. We report the num- ber of targets covered during an episode; higher values indicate better performance. These metrics directly measure completion of the under- lying task rather than the rewards used for optimization. C. Learning Algorithms We instantiate MATES with Independent Proximal Pol- icy Optimization (IPPO) [5] and Independent Soft Actor– Critic (ISAC) [6], as implemented in BenchMARL [7]. IPPO is an on-policy actor–critic method based on PPO, whereas ISAC is an off-policy, entropy-regularized actor– critic method based on SAC. Evaluating both allows us to test whether observation-side transfer remains effective under distinct optimization and data-reuse regimes. We evaluate POGEMA with IPPO and evaluate Navigation and Discovery with both IPPO and ISAC. In every setting, the solo policy and corresponding conventional MARL pol- icy use the same network architecture. The solo policy is trained using the single-agent version of the same learning algorithm, while multi-agent policies share parameters across homogeneous agents. D. Methods and Baselines MATES variants. We evaluate MATES together with MATES-NH, its no-head variant. Both methods freeze the transferred solo actor and critic and train observation adapters during the multi-agent phase. MATES additionally trains the critic’s value head, whereas MATES-NH keeps this head frozen. Their precise architectures and numbers of trainable parameters are reported in Section V-E. Direct training and transfer baselines. We compare against three direct baselines. Solo deploys the pretrained single-agent policy independently for every agent, without access to neighboring-agent information or subsequent multi- agent training. Classic MARL trains the complete parameter- shared policy from random initialization directly on the multi-agent task. Full fine-tuning expands the input layer IPPO ISAC 02.00M4.00M Environment steps 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Throughput (goals/step) 02.50M5.00M7.50M10.00M Environment steps 50 75 100 125 150 175 200 Episode length (steps) 02.50M5.00M7.50M10.00M Environment steps 2 4 6 8 10 12 Targets covered no ISAC implementation 02.50M5.00M7.50M10.00M Environment steps 50 75 100 125 150 175 200 Episode length (steps) 02.50M5.00M7.50M10.00M Environment steps 0.0 2.5 5.0 7.5 10.0 12.5 15.0 Targets covered POGEMA ↑ higher is better · trained at n=64 Navigation ↓ lower is better · trained at n=8 Discovery ↑ higher is better · trained at n=8 Baselines (Training) — task completion solo classic MARL training MATES (ours) finetune (after solo) MATES-NH (ours) (a) Direct reinforcement-learning baselines. IPPO ISAC 02.00M4.00M Environment steps 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Throughput (goals/step) 02.50M5.00M7.50M10.00M Environment steps 50 75 100 125 150 175 200 Episode length (steps) 02.50M5.00M7.50M10.00M Environment steps 0 2 4 6 8 10 12 Targets covered no ISAC implementation 02.50M5.00M7.50M10.00M Environment steps 50 75 100 125 150 175 200 Episode length (steps) 02.50M5.00M7.50M10.00M Environment steps 0 2 4 6 8 10 12 14 Targets covered POGEMA ↑ higher is better · trained at n=64 Navigation ↓ lower is better · trained at n=8 Discovery ↑ higher is better · trained at n=8 Baselines (Training) — task completion solo PegMARL (expert-guided, from scratch) MATES (ours) R2BC (round-robin BC, shared policy) MATES-NH (ours) (b) Expert-guided baselines. Fig. 2: Task completion during training. MATES and MATES-NH are compared with (a) direct reinforcement-learning baselines and (b) expert-guided baselines. Dashed vertical lines mark the transition from solo pretraining to multi-agent training. Curves and shaded regions report the mean and 95% confidence interval over 30 held-out evaluation episodes, respectively. of the pretrained solo network to accept the multi-agent observation, transfers the pretrained weights and optimizes the complete network during multi-agent training. Expert-guidedbaselines. We additionally compare against PegMARL [13] and R2BC [12], both of which learn from expert demonstrations. PegMARL trains behavior IPPO ISAC 8163264128256 Number of agents (team size) 0.5 1.0 1.5 2.0 2.5 Throughput (goals/step) 1248162432 Number of agents (team size) 25 50 75 100 125 150 175 200 Episode length (steps) 248162432 Number of agents (team size) 0 5 10 15 20 25 Targets covered no ISAC implementation 1248162432 Number of agents (team size) 25 50 75 100 125 150 175 200 Episode length (steps) 248162432 Number of agents (team size) 0 10 20 30 40 Targets covered POGEMA ↑ higher is better Navigation ↓ lower is better Discovery ↑ higher is better Baselines (Eval) — task completion solo classic MARL training MATES (ours) finetune (after solo) MATES-NH (ours) PegMARL (expert-guided, from scratch)R2BC (round-robin BC, shared policy) Fig. 3: Generalization across team sizes. Policies trained with 64 agents in POGEMA and 8 agents in Navigation and Discovery are evaluated without further training at the team sizes shown. Points and shaded regions report the mean and 95% confidence interval over 30 held-out episodes, respectively. TABLE I: Parameters optimized during multi-agent training. Percentages are relative to the corresponding fully trainable network. MATES-NH denotes MATES without value-head training. MethodPOGEMA Navigation IPPO Discovery IPPO Navigation ISAC Discovery ISAC Classic MARL121,542 (100%)142,602 (100%)149,258 (100%)214,538 (100%)224,522 (100%) MATES8,931 (7.3%)5,029 (3.5%)8,383 (5.6%)8,189 (3.8%)13,220 (5.9%) MATES-NH1,186 (1.0%)4,772 (3.3%)8,126 (5.4%)7,675 (3.6%)12,706 (5.7%) and transition discriminators on demonstrations from the same solo expert used by MATES and uses their outputs to shape the agents’ rewards. Its original formulation employs agent-specific discriminators and policies together with a shared team reward. We instead share the discriminators and policy parameters across homogeneous agents and apply the shaping signal to the environments’ native per-agent rewards. Besides matching the setting used by all other methods, parameter sharing permits the resulting policy to be evaluated at team sizes different from the training team size. R2BC is a behavior-cloning baseline in which a frozen expert controls agents in round-robin order to provide online action labels for a newly initialized learner. We use the corresponding Classic MARL policy as the expert, rather than the MAPPO expert used in the original implementation, and train a single parameter-shared actor to reflect the homogeneous-agent setting. E. Architectures and Parameter Efficiency All experiments use decentralized, parameter-shared net- works: one network is shared by every homogeneous agent, and observation dimensionality does not grow with team size. For Navigation and Discovery, we use the standard Bench- MARL [7] backbones: IPPO uses separate two-layer width- 256 actor and critic MLPs, while ISAC uses the same actor together with twin two-layer width-256 Q-networks. For POGEMA, we follow the backbone architecture of Follower [21], using a shared 64-channel ResNet encoder with one residual block. CNN components use ReLU activations, whereas the VMAS MLPs use Tanh. The MATES adapters are substantially smaller than these backbones. POGEMA uses a two-layer CNN adapter with 32 hidden channels. In VMAS, each adapter is a one-hidden- layer width-64 MLP: IPPO uses independent adapters for its actor and critic, while ISAC uses one actor adapter and an independent joint observation–action adapter for each Q- network. As specified in Algorithm 1, MATES freezes the trans- ferred single-agent parameters θ before multi-agent training and optimizes only the newly introduced adapter parameters φ. We apply the same principle to the transferred critic. In the IPPO instantiation, MATES also updates the critic’s existing value head, whereas MATES-NH keeps this head frozen and trains only the adapters. Table I quantifies the parameter efficiency of MATES by comparing the number of parameters optimized during multi- agent training with full-network optimization. Following IPPO ISAC 8163264128256 Number of agents (team size) 0.0 0.5 1.0 1.5 2.0 2.5 Throughput (goals/step) 1248162432 Number of agents (team size) 25 50 75 100 125 150 175 200 Episode length (steps) 248162432 Number of agents (team size) 0 5 10 15 20 25 Targets covered no ISAC implementation 1248162432 Number of agents (team size) 25 50 75 100 125 150 175 200 Episode length (steps) 248162432 Number of agents (team size) 0 5 10 15 20 25 Targets covered POGEMA ↑ higher is better Navigation ↓ lower is better Discovery ↑ higher is better Ablations (Eval) — task completion solo adapter only MATES (ours) adapter (frozen random init) MATES-NH (ours) random policy Fig. 4: Ablation results across team sizes. Points and shaded regions report the mean and 95% confidence interval over 30 held-out evaluation episodes, respectively. Algorithm 1, MATES freezes the transferred backbone and trains only the adapters and, where applicable, the critic’s value head. Relative to the parameter count of the corresponding full MARL network, the number of parameters optimized by MATES is 3.5%–7.3%, while that of MATES-NH is 1.0%– 5.7% across all evaluated settings. Thus, MATES requires a substantially smaller trainable parameter budget than full- network optimization. F. Results Performance, parameter efficiency, and generalization. Figure 2 compares task performance throughout training. MATES and MATES-NH perform competitively with Classic MARL, full fine-tuning, PegMARL, and R2BC across the evaluated tasks and learning algorithms. No baseline consis- tently dominates across all settings. This performance does not require optimizing the trans- ferred backbone. As shown in Table I, the trainable parameter count of MATES is equivalent to only 3.5%–7.3% of the corresponding full-network parameter count. Figure 3 evaluates the final policies across team sizes without additional training. Although MATES is trained with 64 agents in POGEMA and 8 agents in Navigation and Discovery, its performance remains competitive as the number of agents increases. Its advantage over the unadapted solo policy generally widens beyond the training team size, and MATES frequently matches or exceeds the fully trained baselines. This is particularly evident in Navigation and Dis- covery under IPPO and in Navigation under ISAC. Together, these results answer RQ1 affirmatively: MATES achieves competitive multi-agent performance with a substantially smaller trainable parameter budget. They also answer RQ2 affirmatively: the learned adaptation remains effective be- yond the team size encountered during training. Contributions of solo competence and observation adaptation. Figure 4 separates the contributions of the transferred solo policy and the learned adapter. The solo policy suc- cessfully completes each task and substantially outperforms the random policy, confirming that it transfers useful task competence. However, its relative performance deteriorates as team size increases, and the gap between the solo policy and MATES generally widens. The adapter-only policy also learns nontrivial behavior but consistently underperforms MATES. When the pretrained solo network is instead replaced by a frozen randomly initialized network, the trained adapter can recover limited performance in some settings, but it does not approach MATES and degrades toward random-policy performance as team size increases. Thus, an arbitrary frozen backbone is insufficient. Together, these results answer RQ3: MATES derives its performance from combining a competent pre- trained policy with an adapter that accommodates multi- agent interaction, rather than from either component alone. Value-head adaptation. Under IPPO, MATES and MATES-NH perform similarly in Navigation and Discovery, indicating that observation adaptation is generally sufficient without updating the transferred value head. Updating the value head provides a slight benefit in POGEMA, particularly at larger team sizes, while having opposite roles across ISAC in the evaluated settings. RQ4 therefore has a qualified answer: value-head adaptation is not generally necessary, although it can improve performance in some settings. VI. CONCLUSION We introduced MATES, a parameter-efficient approach for transferring single-agent policies to multi-agent tasks. MATES preserves the pretrained actor and critic while learning compact input-side adapters that account for multi- agent interaction. Across both discrete and continuous en- vironments instantiated with on-policy IPPO and off-policy ISAC, MATES performs competitively with full MARL training, full fine-tuning, and expert-guided baselines while optimizing a parameter count equivalent to only 3.5%– 7.3% of the corresponding full-network size. The learned adapters also remain effective when deployed at team sizes not encountered during training. Our ablations show that these results depend on both components of the method: solo pretraining supplies useful task competence, while ob- servation adaptation enables that competence to scale to unseen increasing counts of multi-agent deployment. Critic- head adaptation provides task-dependent benefits but is not consistently required. ACKNOWLEDGMENT OpenAI ChatGPT assisted with framing and language in this work, and Cursor (mainly Claude and Grok) assisted with generating the code, edit visualization of the plots, verifying implementation and end-to-end correctness via comprehensive testing. The ideas, algorithms, experiments, results, and implementation are entirely the authors. The authors retain responsibility for all content. REFERENCES [1] S. V. Albrecht, F. Christianos, and L. Schäfer, Multi-Agent Reinforce- ment Learning: Foundations and Modern Approaches. MIT Press, 2024. [2] A. Skrynnik, A. Andreychuk, A. Borzilov, A. Chernyavskiy, K. Yakovlev, and A. Panov, “POGEMA: A benchmark platform for cooperative multi-agent pathfinding,” in International Conference on Learning Representations, 2025. [3] M. Bettini, R. Kortvelesy, J. Blumenkamp, and A. Prorok, “VMAS: A vectorized multi-agent simulator for collective robot learning,” in International Symposium on Distributed Autonomous Robotic Systems (DARS), Springer, 2022. [4] Y. Du, J. Z. Leibo, U. Islam, R. Willis, and P. Sunehag, “A review of cooperation in multi-agent learning,” CoRR, vol. abs/2312.05162, 2023. [5] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of PPO in cooperative multi- agent games,” in Advances in Neural Information Processing Systems, vol. 35, p. 24611–24624, 2022. [6] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” in International Conference on Machine Learning, vol. 80 of Proceedings of Machine Learning Research, p. 1861–1870, 2018. [7] M. Bettini, A. Prorok, and V. Moens, “BenchMARL: Benchmarking multi-agent reinforcement learning,” Journal of Machine Learning Research, vol. 25, no. 217, p. 1–10, 2024. [8] M. E. Taylor and P. Stone, “Transfer learning for reinforcement learning domains: A survey,” J. Mach. Learn. Res., vol. 10, p. 1633– 1685, 2009. [9] F. L. da Silva and A. H. R. Costa, “A survey on transfer learning for multiagent reinforcement learning systems,” J. Artif. Intell. Res., vol. 64, p. 645–703, 2019. [10] M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba, “End to end learning for self-driving cars,” 2016. [11] P. E. Moraes, C. Peters, H. Sodre, W. Moraes, S. Barcelona, J. Deniz, V. Castelli, B. Guterres, and R. B. Grando, “Behavior cloning for mini autonomous car path following,” CoRR, vol. abs/2410.07209, 2024. [12] C. Mattson, V. Raveendra, E. Novoseller, N. Waytowich, V. J. Lawh- ern, and D. S. Brown, “R2BC: Multi-agent imitation learning from single-agent demonstrations,” in IEEE International Conference on Robotics and Automation (ICRA), 2026. [13] P. Yu, M. Mishra, A. Koppel, C. Busart, P. Narayan, D. Manocha, A. S. Bedi, and P. Tokekar, “Beyond joint demonstrations: Personalized expert guidance for efficient multi-agent reinforcement learning,” Transactions on Machine Learning Research, 2025. [14] N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C. Chan, W. Chen, J. Yi, W. Zhao, X. Wang, Z. Liu, H. Zheng, J. Chen, Y. Liu, J. Tang, J. Li, and M. Sun, “Parameter-efficient fine-tuning of large-scale pre-trained language models,” Nat. Mac. Intell., vol. 5, no. 3, p. 220–235, 2023. [15] Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang, “Parameter- efficient fine-tuning for large models: A comprehensive survey,” CoRR, vol. abs/2403.14608, 2024. [16] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” CoRR, vol. abs/1902.00751, 2019. [17] J. O. Zhang, A. Sax, A. Zamir, L. J. Guibas, and J. Malik, “Side- tuning: A baseline for network adaptation via additive side networks,” in Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I (A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, eds.), vol. 12348 of Lecture Notes in Computer Science, p. 698–714, Springer, 2020. [18] T. Silver, K. R. Allen, J. Tenenbaum, and L. P. Kaelbling, “Residual policy learning,” CoRR, vol. abs/1812.06298, 2018. [19] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine, “Residual reinforcement learning for robot control,” in 2019 International Conference on Robotics and Automation (ICRA), p. 6023–6029, 2019. [20] J. Zhang, L. Tai, P. Yun, Y. Xiong, M. Liu, J. Boedecker, and W. Burgard, “Vr-goggles for robots: Real-to-sim domain adaptation for visual control,” IEEE Robotics Autom. Lett., vol. 4, no. 2, p. 1148– 1155, 2019. [21] A. Skrynnik, A. Andreychuk, M. Nesterova, K. Yakovlev, and A. Panov, “Learn to follow: Decentralized lifelong multi-agent pathfinding via planning and learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, p. 17541–17549, 2024.