Paper deep dive
Self-Referenced Social Preferences: Cooperation without Observing Others Rewards
Mohamed Ayman Mohamed, Harshil Kotamreddy, Marcos Menon Jose
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 10/9/2026, 7:26:06 AM
Summary
This paper introduces 'self-referenced social preferences' for multi-agent reinforcement learning (MARL), enabling agents to learn cooperative behavior without observing peers' private reward signals. Agents model their own reward function and apply it to observed transitions of other agents to estimate their outcomes. These estimates are integrated into social preferences via two methods: Self-Referenced Reward (SRR), which modifies the learning reward, and Self-Referenced Advantage (SRA), which weights policy updates. The approach is evaluated on sequential social dilemmas (Escape Room, Clean Up, Commons Harvest), demonstrating that agents can achieve cooperation and equitable outcomes comparable to or better than methods with access to true rewards.
Entities (11)
Relation Signals (10)
Mohamed A. Mohamed ā affiliatedwith ā Amazon
confidence 99% Ā· Mohamed A. Mohamed Affiliation: Amazon, Vancouver, Canada
Marcos Jose ā affiliatedwith ā ItaĆŗ Unibanco
confidence 99% Ā· Marcos Jose Affiliation: ItaĆŗ Unibanco, SĆ£o Paulo, Brazil
Harshil Kotamreddy ā affiliatedwith ā NVIDIA
confidence 99% Ā· Harshil Kotamreddy Affiliation: Nvidia, Santa Clara, United States
Self-Referenced Social Preferences ā evaluatedon ā Escape Room
confidence 99% Ā· We evaluate the approach on three sequential social dilemmas, Escape Room, Clean Up, and Commons Harvest...
Self-Referenced Social Preferences ā evaluatedon ā Clean Up
confidence 99% Ā· We evaluate the approach on three sequential social dilemmas, Escape Room, Clean Up, and Commons Harvest...
Self-Referenced Social Preferences ā evaluatedon ā Commons Harvest
confidence 99% Ā· We evaluate the approach on three sequential social dilemmas, Escape Room, Clean Up, and Commons Harvest...
Self-Referenced Social Preferences ā implementsusing ā Proximal Policy Optimization (PPO)
confidence 97% Ā· Our agents use proximal policy optimization (PPO) (Schulman et al., 2017)...
Self-Referenced Social Preferences ā uses ā
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Social preferences can promote cooperation in multi-agent reinforcement learning, but existing approaches often require agents to observe the rewards of their peers. In many real-world interactions, however, an agent can, as humans do, observe others' behavior and outcomes without access to their private reward signals. We introduce self-referenced social preferences, in which each agent learns a model of its own reward, applies it to other agents' observed transitions to assess their outcomes from its own perspective, and feeds these self-referenced assessments into standard social preferences. We study two ways to incorporate these assessments: modifying the learning reward, or using them to weight policy updates. We evaluate the approach on three sequential social dilemmas, Escape Room, Clean Up, and Commons Harvest, which require volunteering, public-good contribution, and resource restraint, respectively. Across all three environments, agents learn cooperative behavior without observing others' rewards, including in settings where independent learners fail to cooperate, and frequently achieve more equitable divisions of jointly produced returns than agents with access to true rewards. The effective integration point depends on the social preference: inequity aversion works best in the reward together with a value look-ahead, whereas a purely benevolent preference benefits from policy-update weighting. Under partial observability, the policy-update approach continues to support cooperation. These results show that explicit access to other agents' reward signals is not necessary for learning cooperative behavior: social preferences can instead be grounded in self-referenced assessments of others' outcomes derived from their observed behavior.
Tags
Links
- Source: https://arxiv.org/abs/2610.07881v2
- Canonical: https://arxiv.org/abs/2610.07881v2
Trouble viewing inline? Open PDF directly ā
Full Text
151,937 characters extracted from source content.
Expand or collapse full text
Self-Referenced Social Preferences: Cooperation without Observing Othersā Rewards Conference: ; ; Mohamed A. Mohamed Affiliation: Amazon, Vancouver, Canada email: mmayman@amazon.com , Harshil Kotamreddy Affiliation: Nvidia, Santa Clara, United States email: hkotamreddy@nvidia.com and Marcos Jose Affiliation: ItaĆŗ Unibanco, SĆ£o Paulo, Brazil email: marcos.jose@itau-unibanco.com.br Ā© none Abstract. Social preferences can promote cooperation in multi-agent reinforcement learning, but existing approaches often require agents to observe the rewards of their peers. In many real-world interactions, however, an agent can, as humans do, observe othersā behavior and outcomes without access to their private reward signals. We introduce self-referenced social preferences, in which each agent learns a model of its own reward, applies it to other agentsā observed transitions to assess their outcomes from its own perspective, and feeds these self-referenced assessments into standard social preferences. We study two ways to incorporate these assessments: modifying the learning reward, or using them to weight policy updates. We evaluate the approach on three sequential social dilemmas, Escape Room, Clean Up, and Commons Harvest, which require volunteering, public-good contribution, and resource restraint, respectively. Across all three environments, agents learn cooperative behavior without observing othersā rewards, including in settings where independent learners fail to cooperate, and frequently achieve more equitable divisions of jointly produced returns than agents with access to true rewards. The effective integration point depends on the social preference: inequity aversion works best in the reward together with a value look-ahead, whereas a purely benevolent preference benefits from policy-update weighting. Under partial observability, the policy-update approach continues to support cooperation. These results show that explicit access to other agentsā reward signals is not necessary for learning cooperative behavior: social preferences can instead be grounded in self-referenced assessments of othersā outcomes derived from their observed behavior. Keywords: Reinforcement Learning, Sequential Social Dilemmas, Cooperation 1. Introduction Cooperation in multi-agent reinforcement learning (MARL) often requires that agents learn about one another. The difficulty is particularly pronounced in sequential social dilemmas (Leibo et al., 2017), where individual incentives can conflict with collective welfare. In Escape Room, some agents must pull a costly lever so that others can leave; in Clean Up, agents must clean a river without direct reward so that apples continue to grow; and in Commons Harvest, agents must leave apples uneaten so that the resource can regenerate. Independent deep reinforcement learners fail to cooperate in these games (Perolat et al., 2017; Hughes et al., 2018; Yang et al., 2020). A common remedy is to give each agent a social preference, a term in its objective that depends on how the other agents are doing. Inequity-averse agents are penalized for earning more or less than their peers (Hughes et al., 2018), agents with a social value orientation weight the othersā outcomes alongside their own (McKee et al., 2020), and prosocial agents add the othersā rewards to theirs (Peysakhovich and Lerer, 2018). However, while each of these increase cooperation, they do so by reading the rewards of the other agents. This is somewhat unsatisfying. A reward is an agentās private signal of how good an experience was, and giving one agent anotherās reward provides information that it could not obtain by interacting with the world. People do not see how rewarding an experience is for someone else. Autonomous vehicles from different manufacturers have no reason to report their objectives to one another. A prosocial agent that relies on the othersā rewards therefore depends on a signal that exists in simulation but rarely outside it, and its success in simulation does not tell us whether cooperation can be learned without that signal. We relax this requirement by allowing agents to receive only the observations and actions of the other agents. Centralized-training methods already provide access to other agentsā observations and actions during training (Lowe et al., 2017; Foerster et al., 2018b); we retain this assumption while withholding their rewards, critics, and parameters. Humans and other animals face the same problem. They observe what others do, but not how rewarding it is for them, and they use empathy to infer anotherās situation by interpreting observable cues through their own experience (Preston and de Waal, 2002; Decety and Jackson, 2004). Psychology separates empathy into two components: cognitive and motivational empathy. Cognitive empathy involves estimating how another is doing, while motivational empathy is the inner drive to care about their welfare and take action to help them (Decety and Jackson, 2004; de Waal, 2008). Social preferences, which come from the study of human behaviour (Fehr and Schmidt, 1999; Fehr and GƤchter, 2000; Bolton and Ockenfels, 2000; Charness and Rabin, 2002; Henrich et al., 2001), model motivational empathy. Existing multi-agent methods that use social preferences do not model cognitive empathy because they directly provide the agent a numeric value that tells them how another agent is doing. One way humans perform cognitive empathy is through simulation, imagining themselves in the otherās situation (Gordon, 1986; Goldman, 2006). We therefore ask: can agents in MARL learn cooperative behavior when they cannot directly use other agentsā rewards within their social preferences, but can instead assess other agents through their observed behavior and outcomes and the agentās own experience? To answer this question, we study MARL in settings where each agent has access only to its own reward, but where the reward for a given observation and action is the same for every agent. An agent can then learn a model of its own reward from its own transitions and apply it to the transitions of another agent. This allows the agent to use its own reward model to inform its social preferences. When reward functions are shared, it can recover the other agentsā reward; when they differ, it instead evaluates the other agentsā situation using the its own reward model. We call this self-reference. The predicted rewards are given to an unchanged social preference, and each policy still acts only on its own observation. We explore two ways to use this prediction: Self-Referenced Reward (SRR), adds the predicted reward to the agentās reward, and Self-Referenced Advantage (SRA), adds the predicted reward to the coefficient of the policy update. In all three games, the best self-referenced learner exceeds plain PPO in both collective return and Nash welfare. In Escape Room and Clean Up it matches the agents that observe the true rewards. With inequity aversion in Clean Up, the self-referenced agents even produce more than the agents that observe the true rewards, and leave the worst-off agent more than twice as much. Commons Harvest is the exception: the agents that use true rewards give the highest collective return. It is the game where our estimates are weakest, because the reward model reads the policyās image features and is dominated by the large beam penalty, and the extra production goes almost entirely to one agent, so the true rewards raise production but not welfare. A high collective return does not mean that every agent is well off. Someone has to pull the lever or clean the river, and they are not designated beforehand. We find that the most productive configurations often leave one agent with almost nothing, so we also report the lowest agent return and select configurations based on both Nash welfare and collective return. Because the same agents meet in every episode, we also give each agent a standing ledger of everyoneās predicted returns across episodes, which discounts the gains of an agent that has been ahead. With this change, the costly role in Escape Room is shared across agents instead of always falling on the same ones. Our contributions are (i) a MARL setting in which agents observe peersā transitions but not their reward signals; (i) SRR, SRA, and a standing ledger, which use each agentās own rewards to infer peer outcomes; and (i) an empirical study showing when self-referenced assessments recover the benefits of true peer rewards, when they do not, and how they affect productivity and fairness across different social preferences, sequential social dilemmas, partial observability, repeated interaction, and heterogeneous private valuations. 2. Background In this section, we introduce the setting, the learning algorithm our agents use, and the social preferences we build on. We use subscripts for agents and superscripts for time: ritr_i^t is the reward of agent i at time t, [z]+=maxā”(z,0)[z]_+= (z,0), and zĀÆāi=1Nā1āājā izj z_-i= 1N-1 _jā iz_j is the average of a quantity over the agents other than i. 2.1. Markov Games and Sequential Social Dilemmas We consider N agents in a general-sum, partially observable Markov game (Shapley1953; Littman1994; Hansen2004). At each time step t, each agent i receives an observation oito_i^t, a function of the state of the environment, chooses an action aita_i^t according to its policy Ļiā(aiā£oi) _i(a_i o_i), and receives a reward ritr_i^t; the next state depends on the actions of all agents. Each agent maximizes its own expected discounted return, ā”[ātĪ³tārit]E [ _tĪ³^tr_i^t ] with Ī³ā[0,1)Ī³ā[0,1), and there is no team reward. In our main experiments every agent observes the full state, arranged around itself; Section 5.2.4 restricts this view, which makes the games partially observable. A sequential social dilemma is a Markov game in which the policies that are individually best for each agent lead to lower returns for all than a cooperative joint policy that a single defector could exploit (Leibo et al., 2017). In Escape Room, a public good is produced only if enough agents pay a private cost at the same time; in Clean Up and Commons Harvest agents share a renewable resource that individual consumption degrades, a common-pool resource problem (Perolat et al., 2017). 2.2. Actor-Critic Learning and PPO Each agent i improves its policy Ļiā(aā£o;Īøi) _i(a o; _i) by the policy gradient (Sutton et al., 1999; Williams, 1992), which weights ālogā”Ļiā(aitā£oit)ā _i(a_i^t o_i^t) by how much better things went after action aita_i^t than expected. Actor-critic methods implement this coefficient using a target function ĪØit _i^t that depends on the action aia_i: (1) [āĪøJ(Īø)=ĻĪø[ā0ā¤tā¤TāĪølnĻĪø(aitā£sit)ā ĪØit|S0=s0]] [ _ĪøJ(Īø)=E_ _Īø [ _0ā¤ tā¤ T _Īø _Īø(a_i^t s_i^t)Ā· _i^t\; |\;S_0=s_0 ] ] In PPO, ĪØit _i^t is the advantage AitA_i^t, which is estimated using GAE (Schulman et al., 2016). Any term in the target function ĪØit _i^t that does not depend on the action can be subtracted from the return without changing the gradient in expectation; it acts as a baseline (Sutton and Barto, 2018). Supplementary Appendix A.1 gives the policy-gradient, baseline, and GAE equations. Our agents use proximal policy optimization (PPO) (Schulman et al., 2017), and each agent runs it independently (de Witt et al., 2020). 2.3. Social Preferences A social preference makes an agentās objective depend on how the other agents are doing, an idea from behavioural economics (Fehr and Schmidt, 1999) and social psychology (Messick and McClintock, 1968). Given to reinforcement learners as an intrinsic reward, such preferences change what independent learners converge to in sequential social dilemmas (Hughes et al., 2018; McKee et al., 2020); Section 6 reviews this work. In a sequential game a single reward says little about who is ahead, so Hughes et al. (Hughes et al., 2018) measure how well agent j is doing by a discounted running sum of its rewards, (2) ejt=Ī³āĪ»rāejtā1+rjt,e_j^t=Ī³ _r\,e_j^t-1+r_j^t, restarted at the start of every episode, and let each agent maximize the return of a social utility, its own reward plus a social term: (3) uit=rit+Fiā(e1t,ā¦,eNt).u_i^t=r_i^t+F_i (e_1^t,ā¦,e_N^t ). With xj=ejtx_j=e_j^t, we use three social terms: FiEIā(x) F_i^EI(x) =Ī±iāxĀÆāi,FiSVOā(x)=Ī±iā(cosā”Ļiāxi+sinā”ĻiāxĀÆāi), = _i\, x_-i, F_i^SVO(x)= _i ( _i\,x_i+ _i\, x_-i ), (4) FiIAā(x) F_i^IA(x) =āĪ±iNā1ājā i[xjāxi]+āĪ²iNā1ājā i[xiāxj]+. =- _iN-1 _jā i[x_j-x_i]_+- _iN-1 _jā i[x_i-x_j]_+. The three terms encode different reasons to care about others. EI (empathetic influence) is pure benevolence: the agent gains when the othersā average outcome rises, as with the prosocial reward of Peysakhovich and Lerer (Peysakhovich and Lerer, 2018). SVO fixes a ratio between oneās own outcome and the othersā through an angle Ļi _i (Messick and McClintock, 1968; McKee et al., 2020): Ļi=0 _i=0 is purely self-regarding and Ļi=Ļ/4 _i=Ļ/4 weights both equally. IA is the inequity aversion of Fehr and Schmidt (Fehr and Schmidt, 1999) and cares about differences rather than levels: it penalizes being behind the others, with weight Ī±i _i, and being ahead of them, with weight Ī²i _i. With IA, Eq. (3) is the learner of Hughes et al. The two IA weights act differently in the dilemmas we study: aversion to being ahead makes an agent share the work that benefits everyone, as cleaning does in Clean Up, whereas aversion to being behind gives it a reason to punish agents that take more, as by firing a beam in Commons Harvest (Hughes et al., 2018). Whatever the reason is, each of these terms needs eje_j for every other agent j, and so the other agentsā rewards. A social term can be implemented within PPO (and other actor-critic methods) in two places: in the reward rir_i, which changes both the advantage and what the critic learns, or in the coefficient AitA_i^t that weights each action, which changes the direction of the update and leaves the reward and the critic unchanged. The baseline property limits what a term can do in the coefficient: a term added to the coefficient changes the expected update only if it depends on the action, otherwise it acts as a baseline. Our two methods modify the update in these two places respectively. Neither is specific to PPO: every reinforcement learning algorithm learns from a reward, and every actor-critic weights its policy gradient by an advantage. We use PPO as a representative actor-critic learner because it is widely used in recent work on sequential social dilemmas (de Witt et al., 2020; Leibo et al., 2021). 3. Problem Formulation The social utility of Eq. (3) requires every agent to know the rewards of all the other agents. Here we define the setting in which an agent knows only its own reward, and show how empathy can be modeled in such a setting. Withholding reward signals. We consider the Markov game of Section 2.1 with N independent learners. During training, every agent i receives the transitions of all agents, with their episode boundaries, but only its own reward: (5) i=(ojt,ajt,ojt+1):j=1,ā¦,N,tā„0āŖrit:tā„0.D_i= \(o_j^t,a_j^t,o_j^t+1):j=1,ā¦,N,\ tā„ 0 \āŖ \r_i^t:tā„ 0 \. Methods with centralized training and decentralized execution (Lowe et al., 2017; Foerster et al., 2018b) and methods that model other agents (Raileanu et al., 2018; Rabinowitz et al., 2018; Jaques et al., 2019) already give agents the othersā observations and actions; the difference here is that the othersā rewards are withheld. The agents share no parameters, critics, or reward models and exchange no messages, and at execution each policy Ļiā(aiā£oi) _i(a_i o_i) acts on its own observation. Modeling empathy. From iD_i, agent i can compute its own trace eie_i but not the traces eje_j of the others, and so it cannot compute FiF_i. It therefore faces two problems. It must estimate, from iD_i alone, a quantity xiājx_ij that can stand in for eje_j, which is the cognitive part of empathy, and it must decide how to use that estimate when it learns, which is the motivational part. An agent that is given the rewards rjr_j computes eje_j exactly, and we use it as the true-reward reference. Estimating the rewards of other agents. Self-reference is most direct when every agentās reward comes from the same reward function R, (6) rjt=Rā”(ojt,ajt,ojt+1)for all āj,r_j^t=R(o_j^t,a_j^t,o_j^t+1) all j, where each observation is arranged relative to its own agent, so that R applies from every agentās position. Under this condition, a model of oneās own reward can approximate the otherās actual reward. The condition holds in our main experiments: a reward comes from an event in the agentās own transition, such as eating an apple, pulling the lever, reaching the open door, or firing or being hit by a beam, and every agent is rewarded for these events in the same way. Agent i can then learn R from its own transitions and apply it to jās, provided that jās transition shows the rewarded event and resembles transitions that i has experienced. Two caveats remain. (1) When a transition does not fully determine the reward, a model fitted by regression estimates its mean, and a nonlinear social term computed from the mean, such as inequity aversion with its [ā ]+[Ā·]_+, need not equal the same term computed from the actual rewards. (2) When the reward functions differ, the estimate is no longer jās reward but the reward that i would receive in jās place; Section 5.2.5 tests this case. 4. Self-Referenced Social Preferences Agent i cannot observe rjr_j, but it observes jās transitions and knows its own rewards. It therefore learns a model of its own reward and applies it to jās transitions, asking what it would have earned in jās place. The key idea is not to reconstruct another agentās private reward exactly, but to evaluate its observed situation using the focal agentās own learned reward model. Figure 1 shows the resulting flow of information inside one agent. These estimates are used in a social term that modifies the update rule in the two ways described in Section 2.3: modifying the reward, which we call Self-Referenced Reward, and modifying the coefficient of the policy update, which we call Self-Referenced Advantage. Supplementary Appendix A.2 gives one training iteration in Algorithm S1. Agent iiAgent jjEnvironmentoit,rito_i^t,r_i^taita_i^tojt,rjto_j^t,r_j^tajta_j^t(ojt,ajt,ojt+1)(o_j^t,a_j^t,o_j^t+1)rjtr_j^t withheldĆ(a) What agent i receivesown transitions(oit,ait,oit+1),rit(o_i^t,a_i^t,o_i^t+1),r_i^tjās transitions(ojt,ajt,ojt+1)(o_j^t,a_j^t,o_j^t+1)rewardmodel fif_iestimatedr^iājt r_ij^tsocial termFi(xiā t)F_i(x_iĀ·^t)ledger GiājG_ij(past episodes)SRR rewarduit=rit+Fiu_i^t=r_i^t+F_icritic ViV_ion oio_i and ojo_jcoefficientĪØit _i^tpolicyĻi _ifitapplyxiājtx_ij^tgatetargetSRA: r^iājt r_ij^tAitA_i^t, A^iājt A_ij^tupdate(b) Inside agent i Figure 1. Self-referenced social preferences. (a) During training agent i receives the transitions of every other agent j, but never jās reward. (b) Agent iās reward model, fitted on its own transitions, assesses jās transitions, giving self-referenced estimates r^iājt r_ij^t, whose running sums xiājtx_ij^t feed an unchanged social term FiF_i. SRR adds the term to the reward; SRA evaluates the critic on jās observations and weights the scores into the policy-update coefficient ĪØit _i^t. The ledger discounts iās gains when i has been ahead.Diagram illustrating self-referenced social preferences, agent observations, and reward modeling. 4.1. Self-Referenced Reward Estimates Agent i fits fiā(o,a,oā²)f_i(o,a,o ) by regression on its own transitions and applies the same model to every observed transition, (7) āfi=12āā¬iā[(fiā(oit,ait,oit+1)ārit)2],r^iājt=fiā(ojt,ajt,ojt+1),L_f_i= 12\,E_B_i [ (f_i(o_i^t,a_i^t,o_i^t+1)-r_i^t )^2 ], r_ij^t=f_i(o_j^t,a_j^t,o_j^t+1), where ā¬iB_i is a minibatch of iās own transitions, and with r^iāit:=rit r_i^t:=r_i^t. In practice, f is learned using a shared representation with the policy. We call r^iāj r_ij a self-referenced reward estimate. If the agents share a reward function, it estimates jās reward; otherwise, it estimates the reward that i would assign to jās transition. Rare rewards create a coverage problem: an agent that only cleans may stop seeing apple rewards. We therefore add a second term to the loss of Eq. (7) over a minibatch drawn from a replay of the agentās own nonzero-reward transitions. The social term is off for the first W environment steps, while the model is fit. The estimates are treated as constants, so no gradient flows from the social term into the reward model. 4.2. Estimating Future Outcomes Agent i keeps one trace for every agent j, computed from its estimates exactly as Eq. (2) computes it from true rewards: (8) eiājt=Ī³āĪ»rā(1āsjt)āeiājtā1+r^iājt,e_ij^t=Ī³ _r(1-s_j^t)e_ij^t-1+ r_ij^t, where sjts_j^t marks the start of an episode. A trace judges an agent only by what it has already earned, so an agent that has just cleaned the river looks worse than the other agents, even though apples will soon grow for it. To amend this, the +V+V variant adds a term that estimates what agent i expects j to earn from in the future. This is done using its critic ViobjV_i obj, estimating the return of iās own objective, evaluated on jās next observation: (9) xiājt=eiājt+wVāĪ³ā(1ādj,tterm)āViobjā(ojt+1),x_ij^t=e_ij^t+w_VĪ³(1-d_j,t term)V_i obj(o_j^t+1), where dj,ttermd_j,t term marks the end of jās episode and wV=(1āĪ³)/(1āĪ³āĪ»r)w_V=(1-Ī³)/(1-Ī³ _r) puts the value on the scale of the trace. The look-ahead is iās value of being in jās position, not an estimate of jās reward. xiā x_iĀ· takes the place of the true traces in the social terms of Eq. (4). 4.3. Self-Referenced Reward (SRR) SRR adds the social term of the self-reference to the agentās reward, (10) uit=rit+Fi(xiā t),u_i^t=r_i^t+F_i(x_iĀ·^t), and PPO computes the advantage and the critic target from uiu_i. With the true rewards in place of r^iāj r_ij, the same construction gives the true-reward learners, which we denote TR; TR-IA without +V+V is the inequity-averse agent of Hughes et al. (Hughes et al., 2018). Using reward estimates in place of the true reward brings one caveat: a poor estimate affects both the policy and the critic. 4.4. Self-Referenced Advantage (SRA) SRA instead uses the estimates in the policy update coefficient (see Eq. (1)). It leaves rir_i and ViobjV_i obj unchanged and changes only the coefficient that weights each action. Agent i evaluates its critic on jās observations and forms a temporal-difference error and its generalized advantage estimate, which we call the cross score, (11) Ī“^iājt Ī“_ij^t =r^iājtāViobjā(ojt)+Ī³ā”(1ādj,tterm)āViobjā(ojt+1), = r_ij^t-V_i obj(o_j^t)+Ī³(1-d_j,t term)V_i obj(o_j^t+1), (12) A^iājt A_ij^t =Ī“^iājt+Ī³āĪ»SRAā(1ādj,tend)āA^iājt+1, = Ī“_ij^t+Ī³ _ SRA(1-d_j,t end) A_ij^t+1, where dendd end also marks a time-limit truncation. The score measures how much better jās transition went than iās critic expected, judged by iās own reward model and values; it is not the advantage of either agentās policy, as the critic was trained on iās observations and objective, not jās. Let qiāiq_i be iās own GAE and qiājq_ij the cross score of Eq. (12), each standardized over time and parallel environments. EI and SVO combine them linearly: (13) ĪØi,EIt=qiāit+Ī±iqĀÆi,āitĪØi,SVOt=(1+Ī±icosĻi)qiāit+Ī±isinĻiqĀÆi,āit. _i,EI^t=q_i^t+ _i q_i,-i^t\; _i,SVO^t=(1+ _i _i)q_i^t+ _i _i q_i,-i^t. The 1 in front of Ī±icosĻi _i _i keeps the agentās own score in the update for every angle Ļi _i. For IA, the levels decide the weights. For jā ijā i, wiājt w_ij^t =Ī²i[xiāit>xiājt]āĪ±i[xiāit<xiājt]Nā1,wiāit=1āājā iwiājt, = _i1[x_i^t>x_ij^t]- _i1[x_i^t<x_ij^t]N-1, w_i^t=1- _jā iw_ij^t, (14) ĪØi,IAt _i,IA^t =wiāitāqiāit+ājā iwiājtāqiājt. =w_i^tq_i^t+ _jā iw_ij^tq_ij^t. Being ahead of j moves weight from iās own score to jās; being behind j puts negative weight on jās score and adds the same weight to iās own. The coefficient, normalized within each minibatch, is used as the target function in Eq. (1), and no gradient flows through it or into agent j. SRA is therefore a cross-agent credit-assignment rule rather than an unbiased policy-gradient estimator of a fixed social objective. EI and SVO thus add up the agentsā outcomes much as in the reward, whereas for IA the comparison only decides how the scores are weighted: nothing in the agentās return penalizes being ahead or behind, as it does in SRR. 5. Experiments 5.1. Experimental Setup We evaluate our methods in three sequential social dilemmas. In Section 5.2 we seek to answer the following questions: (1) Do agents cooperate without observing each otherās rewards, (2) where should the self-referenced estimate be used in the learning update, (3) which agents take a penalty for cooperation to occur, and (4) whether self-referenced estimates are still useful when the environment is partially observable or rewards are heterogeneous . 5.1.1. The Three Games The games require different behavior for cooperation to occur: volunteering, labor, and restraint. In the main experiments, all three games are fully observable, so the only thing an agent lacks is the other agentsā rewards. Every observation is arranged around its own agent, which is what lets an agent apply its own reward model to another agentās observation; Supplementary Table S2 lists what each agent observes and can do. Escape Room. ER(N,M) is the Escape Room environment introduced by Yang et al. (Yang et al., 2020). At every step, each of N agents chooses stay, lever, or door. Pulling the lever results in ā1-1 reward; if at least M=āN/2āM= N/2 agents pull the lever in the same step, the door opens and every agent at the door earns +10+10. Episodes last at most five steps. The group optimum is 10ā(NāM)āM10(N-M)-M per episode (ex. 17 with 55 agents). Selfish learners earn 0 reward since pulling the lever only ever pays somebody else. The optimum concentrates the cost on the M agents at the lever unless they take turns over many episodes. Clean Up. Clean Up (Hughes et al., 2018) is a gridworld public-goods game: agents earn +1+1 per apple eaten, apples stop growing when waste fills the river, and cleaning the river restores growth but pays the cleaner nothing. We use a 25Ć1825Ć 18 map with 1,000-step episodes. Cooperation requires that some agents clean while the others eat, so the work and the apples can be divided very unevenly. The main results use five agents; Supplementary Tables S9 and S10 report 3 and 7 agents respectively. Commons Harvest. Commons Harvest (Perolat et al., 2017) is a common-pool resource game on a 16Ć3816Ć 38 map with seven agents and 1,000-step episodes. Eating an apple is worth +1+1. Apples regrow in empty cells with a probability p at each time step. p is 0 if there are no apples around the cell and p is larger when there are more apples around it. Thus, eating the last apples of a patch destroys future apples for everyone. Each agent has a beam that allows it to give other agents penalties. The agent that shoots the beam receives a reward of -1 and the agent it hits receives -50. 5.1.2. The Learning Agents Every agent is an independent PPO learner with its own actor, critic and, for self-referenced agents, reward model; nothing is shared between agents. We train agents in Escape room for 500k environment steps and agents in the Clean Up and Commons Harvest environments for 5M steps. In the partially observable setting, we train agents for 7M steps. Supplementary Appendix B.2 lists the networks, warm-up, and PPO settings. We compare four families of learners. Two are baselines: plain PPO, which has no social term, and the true-reward learners, TR, which add the operators of Eq. (4), computed from traces of the other agentsā true rewards, to the reward; TR-IA is the agent of Hughes et al. (Hughes et al., 2018). The true-reward learners act as a comparison to show which settings benefit from providing agents with access to the true reward. SRR adds the same operators, computed from self-referenced estimates, to the reward, with or without the look-ahead +V. Without the look-ahead it differs from TR only in the source of the rewards and in the warm-up of its reward model. The main figures show the reward learner with the look-ahead, SRR+V; Section 5.2.2 compares both. SRA puts the estimates in the coefficient of the policy update and leaves the reward and the critic unchanged. Each family comes with the EI, SVO, and IA operators, with weights tuned for each game and group size, and we name a learner by family and operator, as in SRR-IA. A self-scaling control, SVO with Ļ=0Ļ=0, uses no information about the others; it tests whether a gain comes from the social term or merely from rescaling the agentās own reward. For the repeated game, we add a standing gate (see Supplementary Appendix) to four selected Escape Room configurations, SRA-EI with three and five agents and SRR-IA with five and seven, and change nothing else. For Section 5.2.5, two of the five Clean Up agents value an apple at 2 instead of 1; we reuse the settings selected under shared rewards and train on the same held-out seeds. Selection and evaluation. We tune on three seeds in every game, and on two for the partial-view retuning, and score every candidate by the mean minus the standard deviation of a metric across the tuning seeds. We choose once by collective return and once by Nash welfare (see Section 5.1.3) and call these the learnerās collective-return and Nash welfare selections. Unless stated otherwise, figures show the collective-return selection. Each choice is frozen and trained from scratch on eight seeds that played no part in tuning, and we never reselect on these runs. Intervals are two-sided Student-t 95% intervals across the eight seeds. Partial view. With the partial view, each agent in Clean Up and Commons Harvest sees only the 15Ć1515Ć 15 window around itself. The transition agent i receives from j then shows only jās neighborhood, so i must judge jās situation from a view it may never have had itself. We retune every learner for this setting, with feed-forward agents and with agents whose policy and critic each carry a GRU (Cho et al., 2014). The grid and the recurrent architecture can be found in Appendix B.4. Figure 2. Collective return over training. Each panel shows plain PPO, the TR learner, and both of our methods: SRA and SRR+V. Lines are means over eight seeds and shaded regions are pointwise 95% t intervals. The dotted line is the Escape Room optimum, 1717. Commons Harvest TR-IA is outside the plotted range, with final mean C=ā1,935C=-1,935, and is marked at the panel edge. 5.1.3. Outcome and Behavioral Measures Because every agent keeps the same index in every episode of a run, we can average each agentās return over many episodes. Let RĀÆi,s R_i,s be agent iās episodic return averaged over the final 20,000 environment steps of seed s in Clean Up and Commons Harvest and over the final window of training in Escape Room. From these N averages we compute (15) Cs=āiRĀÆi,Ls=miniā”RĀÆi,āc,s=e1Nāāilogā”(1+[RĀÆi+c]+)ā1. C_s= _i R_i,\;L_s= _i R_i,\;NW_c,s=e 1N _i (1+[ R_i+c]_+)-1. The collective return CsC_s is how much the group produces, whoever receives it; the lowest agent return LsL_s shows whether some agent is left behind; and the Nash welfare āc,sNW_c,s (Kaneko and Nakamura, 1979; Caragiannis et al., 2019), a geometric mean that for a fixed total is largest when the returns are equal, rewards both production and balance, with a shift c of 5 in Escape Room and 0 elsewhere. Because returns are averaged over many episodes, these measures can tell agents that take turns at a costly role from agents whose roles stay fixed. In Escape Room, we also report each agentās exit rate, the fraction of episodes in which it reaches the open door, and call the lever role shared on a seed when every agent has a positive return and the exit rates have a standard deviation below 0.20. In Commons Harvest, we report sustainability (Perolat et al., 2017), the mean time step at which apples are eaten; higher means agents leave apples for later. 5.2. Results 5.2.1. Can Agents Cooperate without Observing Othersā Rewards? In all three games, social preferences can operate on self-referenced assessments without observing the othersā rewards. How closely those assessments match the true rewards varies by game, but exact reward recovery is not necessary for cooperation. Figure 2 compares plain PPO, the true-reward learners, and both of our methods for every game and operator. Escape Room is the most straightforward test, because pulling the lever only ever benefits someone else, and plain PPO agents never learn to pull it. A self-referenced agent can still see the benefit: it has learned from its own exits that reaching the open door pays ten, so when another agent stands at the door, it recognises that reward. Thus, SRA-EI and SRR-IA+V reach the optimum, as the agents given the true rewards do. SVO on the other hand weighs the agentās own outcome as well as the othersā. Thus, the benefit to other agents must outweigh the agentās own cost of pulling the lever. With true rewards and a strong weight it does, and the agents learn to pull. SRR-SVO was selected with a much weaker weight, because that weight was already enough on the tuning seeds; with it, pulling still feels like a loss, and cooperation rests on the look-ahead alone, so it reaches only about half the optimum. SRA-SVO caps how much weight it can give to the others (see Section 5.2.2). Supplementary Appendix C.4 shows that at least one self-referenced learner reaches the optimum from two to twelve agents. Clean Up requires agents to sacrifice reward to provide labor. Cleaning the river earns nothing directly, and plain PPO agents rarely do it. Every self-referenced learner produces about as much as its true-reward counterpart, so self-referenced estimates are enough to sustain a public good. Under inequity aversion, SRR even does better than TR: SRR-IA+V produces more than TR-IA and more than doubles the return of its poorest agent. We think the true rewards are the obstacle. An agent that has just eaten an apple seems far ahead of one that is cleaning, although apples are about to grow for the cleaner. With a strong penalty on being ahead, agents given the true rewards stop eating, and TR-IA harvests nothing on any tuning seed, so its selection settles on a weak penalty. Commons Harvest requires agents to restrain themselves. Agents must leave apples uneaten so that patches regrow, and an agent that eats the last apples of a patch takes future apples from everyone. SRA-EI learns this restraint without seeing the othersā rewards. It doubles plain PPOās collective return, and its agents eat apples four times later on average. This is due to the policy update. When an agent strips a patch, its critic values the other agentsā positions less turning their cross scores turn negative. Thus, the action that stripped the patch loses credit. This is also the game where our self-referenced learners fall furthest behind the true rewards, and TR-EI and TR-SVO produce more still. Estimation is hardest here, as the reward model is learned through a shared representation with the policy, whose image features were not trained to recognize rewards, and the beamās large penalty dominates what it learns. Inequity aversion goes the other way. An agent averse to being behind has a reason to fire at an agent that is ahead, and with true rewards that gap is visible at every step, so the agents of TR-IA almost stop harvesting or fire at one another (Hughes et al., 2018). SRR-IA+V was selected with a weaker penalty on being behind, so it has little reason to fire, and it keeps harvesting. Taken together, self-referenced assessments are enough to start and sustain cooperation in all games. But, in Commons Harvest, the true-reward learners produce more. In 5.2.3, we look at the distribution of return among agents when using each of these methods. Figure 3. Who receives the return and who does the work in Clean Up with five agents. Top: every agentās return over training; agents are ranked within each seed by their final return and averaged by rank over the eight seeds, darkest the richest. Bottom: each agentās final share of the apples, solid, and of the cleaning, dashed, with 95% confidence intervals; 20% is an equal share. 5.2.2. Where Should the Estimate Enter? The two methods of self-reference suit different kinds of social preference. Inequity aversion works only with SRR: SRR-IA+V is the better of our methods in every game, while SRA-IA never escapes Escape Room. In the reward, being ahead costs the agent part of its return, which it can learn to avoid. In the policy update, being ahead only shifts weight between scores, (see Eq. (14)). Thus, nothing pushes it to stop being ahead. EI only adds up the othersā outcomes, which credits actions with the gains of other agents. Thus, SRA-EI is the better method in Escape Room and Commons Harvest and about as good in Clean Up. SVO also adds outcomes but weighs the agentās own as well. In the policy update this caps the weight assigned to the other agents at tanā”Ļi _i times the agentās own weight, which is not enough to achieve the performance of EI.Thus, we note that a preference comparing agents belongs in the reward. A benevolent pereference such as EI is safer in the policy update. The look-ahead is what makes SRR work. Without it, agents that never reach the door never see a reward to assess, so the social term stays at zero. The criticās value distinguishes positions before any reward is assessed and gets cooperation started. Under inequity aversion, it also prevents the agent misjudging the cleaner.Under EI, the look-ahead brings no clear gain outside Escape Room: in Clean up, SRR-EI+V produces about 290 apples less than SRR-EI, and in Commons Harvest, it stays at plain PPOās level, far below SRR-EI. Group size exposes the limit of SRA. Its coefficient averages over the Nā1N-1 others, so any one agentās action matters less as the group grows, and SRA-EI falls further behind TR-EI. SRR, on the other hand, does not dilute, and here self-reference again holds up better TR: IA with true rewards declines as the group size grows, while with self-referenced estimates and the look-ahead it holds. A larger group gives an inequity-averse agent more pairs to be behind in, and true rewards make each comparison sharper than it is. Appendix D.2 gives the results with 2ā122-12 agents. Figure 4. Partial view: collective return over training in Clean Up and Commons Harvest with a 15Ć1515Ć 15 window, for feed-forward and GRU agents. Lines are means over eight held-out seeds and shaded regions pointwise 95% Student-t intervals. 5.2.3. Who Pays for Cooperation? Collective return on its own does not give us the full picture. Figure 3 shows which agents gained the most return and which agents cleaned the most in Clean Up. It shows the most productive TR learner and self-referenced learner for each operator; further details can be found in Appendix E.1. The learner that produces the most shares it worst. TR-EI produces 1,530 apples, but its poorest agent earns nothing for the whole run, and the bottom row shows why: the richest agent eats a third of the apples and cleans almost nothing, while the two poorest do 96% of the cleaning. The group has split into eaters and cleaners, and collective return counts both the same. Nash welfare does not: as a geometric mean, it equals the average return when every agent earns the same and falls towards zero when any agent is left with nothing, so TR-EIās is only 70, although its agents average about 300 apples. SRA-EI and SRA-SVO soften the split, raising the poorest agent to 83 and 84, but their two poorest still do most of the cleaning. SRR-IA+V is the exception. All 5 agentsā returns climb together, every agent takes about a fifth of the apples and a fifth of the cleaning, and its Nash welfare, 270, equals its average return, at a collective return about 12% below TR-EIās. Its high welfare comes from sharing the work, not from every agent earning equally little. For every operator, the best self-referenced learner divides the return more evenly than the true-reward learner with the same operator, and across the whole Clean Up tuning grid, selecting on collective return picks a true-reward learner while selecting on Nash welfare picks one of ours. What spreads the return is not the self-referenced estimate alone but where it is used: the reward learners with EI and SVO use the same estimates as SRA, yet split into eaters and cleaners just as the true-reward learners do. The other games show the same tension in their own form. In Commons Harvest the extra apples of the true-reward learner go almost entirely to one agent, leaving its Nash welfare below that of plain PPO, although welfare there needs care, since a control that ignores the others scores highest.In Escape Room every learner that escapes leaves the same agents paying for the lever in every episode. A standing ledger of self-referenced returns across episodes spreads this cost, at about a quarter of the optimum, because nothing tells the agents who should pay this time (See Supplementary Appendix F). 5.2.4. What if Agents See Only Part of the Map? With a partial view the assessment itself becomes harder: the transition that agent i receives from j shows only jās neighbourhood, a part of the map i may never have seen from there. Figure 4 shows Clean Up and Commons Harvest with this window. SRA keeps its cooperation. SRA-EI holds its full-view production in Clean Up and still produces several times as much as plain PPO in Commons Harvest. Memory helps in Clean Up, where its recurrent agents produce more and divide the return best of all learners, while the true-reward learner again leaves its poorest agent with nothing.Inequity aversion in the reward collapses, and the reason is instructive. The estimates are not to blame; they are as accurate as with the full view. The look-ahead is. Removing it partly restores cooperation, and adding it to agents given the true rewards destroys theirs too. The look-ahead reads the criticās value off a window that shows a different part of the map, and an agent averse to being ahead reacts to a gap that is not there. The component that rescued inequity aversion under the full view is the one that breaks it when agents see less, while in the policy update an error adds noise to the credit. Appendix G gives the details. 5.2.5. What if Agents Value Things Differently? We now violate the condition under which self-reference can recover the otherās reward. Two of five Clean Up agents value an apple at 2 rather than 1, so a self-referenced agent cannot recover the otherās true reward from its transition: it assigns the transition the value that it would itself receive. Self-reference remains useful nonetheless: for every operator, our learner divides the return more evenly than the true-reward learner and leaves its poorest agent better off, at a similar collective return. What no learner does, with true or self-referenced rewards, is divide by value. The high-value agents eat about as many apples as the others, so they end with about twice the return whoever is learning. Whether self-reference misleads agents that would otherwise act on values, as people are misled when they assume others value what they value (Ross et al., 1977; Epley et al., 2004), remains open. Supplementary Appendix H gives the details. 6. Related Work Social preferences without reward access. Social preferences such as inequity aversion (Hughes et al., 2018), social value orientation (McKee et al., 2020), and prosocial rewards (Peysakhovich and Lerer, 2018) directly use other agentsā rewards or outcomes. Forward-looking social utilities provide another way to use these outcomes: AFP combines altruistic and fairness preferences through a CES utility applied to expected future returns (Wei et al., 2026). Unlike our method, it still requires access to other agentsā rewards; we instead replace the peer-reward input with a self-referenced estimate. Related approaches reshape incentives through reward transfer or learned incentives (Lupu and Precup, 2020; Yang et al., 2020; Willis et al., 2024). Social influence (Jaques et al., 2019) also avoids access to othersā rewards by rewarding causal influence over their actions, including in a decentralized variant that learns a model of other agents from observed actions. Unlike our method, however, influence is the social signal itself rather than a representation of the otherās outcome. We keep the social preferences unchanged and replace only their input. Modeling others from oneās own experience. Empathy is often described in terms of taking anotherās perspective, by projecting oneself into their situation and using oneās own representations to understand their state (Preston and de Waal, 2002). This intuition is closely related to Simulation Theory, which proposes that people can understand othersā mental states by simulating those states using their own cognitive mechanisms (Gordon, 1986). In multi-agent learning, agents have inferred othersā goals with their own policy (Raileanu et al., 2018), evaluated their own value function from anotherās perspective (Bussmann et al., 2019), and used perspective-taking to decide how much reward to give away (Kong et al., 2024). We use the same idea for standard social preferences and ask where the estimate should enter learning. Inverse RL (Ng and Russell, 2000; Natarajan et al., 2010) and inverse planning (Baker et al., 2009) infer anotherās reward or goals from its behavior under an assumed decision model; self-reference instead assesses the otherās situation with the agentās own reward model, without requiring the other agent to act optimally. Unlike methods that keep rewards private but exchange action suggestions (Jin et al., 2025), our agents exchange no messages. Other approaches learn social reward models from external evaluations of collective outcomes (Caftory et al., 2026). Our agents receive no such external evaluation and instead construct social signals solely from their own reward experience. Appendix J extends this review. 7. Conclusion and Future Work Social preferences do not require agents to directly observe each otherās rewards. An agent can instead assess another agentās situation from its own experience, by applying its learned reward model to the otherās observed transitions. When reward functions are shared, this assessment can approximate the otherās realized reward; when preferences differ, it instead provides a self-referenced counterfactual assessment. In both cases, the resulting signal can sustain cooperation in sequential social dilemmas. Several directions remain open. Agents whose rewards come from different events are beyond a model of oneās own reward, and an agent could instead keep its own model as a prior and correct it from what the others do, combining self-reference with inverse reinforcement learning. SRR needs no actor-critic, so it could carry self-referenced preferences to value-based learners. Letting policies see the record of past episodes, or communicate, could turn sharing into deliberate turn-taking (Axelrod, 1984). A look-ahead that judges another agentās situation from memory rather than from a single window could make SRR robust to partial views. Finally, the setting fits mixed groups of people and machines, where an agent must cooperate with others whose rewards it can never observe. 8. Acknowledgment We would like to thank Rafael M. Cheang for the discussions that led to the initial conception of the ideas presented in this paper, as well as for his valuable insights and help with their formulation. We are also grateful to Prof. James R. Wright, whose course, Modelling Human Strategic Behaviour, provided the setting for the first draft of this work. His feedback during the course, helped shape, expand, and formalize the ideas presented here. Finally, we thank Prof. Marlos C. Machado for his continued support and valuable feedback, as well as for providing computational resources that enabled us to further develop and finalize this project. References Agapiou et al. (2023) J. Agapiou, A. S. Vezhnevets, E. DuƩƱez-GuzmĆ”n, J. Matyas, Y. Mao, P. Sunehag, R. Kƶster, U. Madhushani, K. Kopparapu, R. Comanescu, D. Strouse, M. Johanson, S. Singh, J. Haas, I. Mordatch, D. Mobbs, and J. Z. Leibo Melting Pot 2.0. External Links: 2211.13746 Cited by: Appendix J. Albrecht and Stone (2018) S. V. Albrecht and P. Stone Autonomous agents modelling other agents: a comprehensive survey and open problems. Artificial Intelligence 258, p. 66ā95. Cited by: Appendix J. Axelrod (1984) R. Axelrod The evolution of cooperation. Basic Books. Cited by: Appendix J, Ā§7. Baker et al. (2009) C. L. Baker, R. Saxe, and J. B. Tenenbaum Action understanding as inverse planning. Cognition 113 (3), p. 329ā349. Cited by: Appendix J, Appendix J, Ā§6. Batson (1991) C. D. Batson The altruism question: toward a social-psychological answer. Lawrence Erlbaum Associates. Cited by: Appendix J. Bolton and Ockenfels (2000) G. E. Bolton and A. Ockenfels ERC: a theory of equity, reciprocity, and competition. American Economic Review 90 (1), p. 166ā193. Cited by: Appendix J, Ā§1. Bussmann et al. (2019) B. Bussmann, J. Heinerman, and J. Lehman Towards empathic deep q-learning. In IJCAI-19 AI Safety Workshop, Vol. 2419, p. 111ā117. Cited by: Appendix J, Ā§6. Caftory et al. (2026) A. Caftory, A. Zemach, M. Butman, and D. Friedman Why study emergent behavior when you can regulate it? aligning multi-agent systems with reward prediction. External Links: 2608.07280 Cited by: Appendix J, Ā§6. Camerer (2003) C. F. Camerer Behavioral game theory: experiments in strategic interaction. Princeton University Press. Cited by: Appendix J. Caragiannis et al. (2019) I. Caragiannis, D. Kurokawa, H. Moulin, A. Procaccia, N. Shah, and J. Wang The unreasonable fairness of maximum Nash welfare. ACM Transactions on Economics and Computation 7 (3), p. 12:1ā12:32. Cited by: Appendix J, Ā§5.1.3. Charness and Rabin (2002) G. Charness and M. Rabin Understanding social preferences with simple tests. The Quarterly Journal of Economics 117 (3), p. 817ā869. Cited by: Appendix J, Ā§1. Cho et al. (2014) K. Cho, B. van MerriĆ«nboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio Learning phrase representations using RNN encoderādecoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 1724ā1734. Cited by: Ā§B.4, Ā§5.1.2. de Waal (2008) F. de Waal Putting the altruism back into altruism: the evolution of empathy. Annual Review of Psychology 59 (1), p. 279ā300. Cited by: Appendix J, Ā§1. de Witt et al. (2020) C. S. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. S. Torr, M. Sun, and S. Whiteson Is independent learning all you need in the starcraft multi-agent challenge?. External Links: 2011.09533 Cited by: Appendix J, Ā§2.2, Ā§2.3. Decety and Jackson (2004) J. Decety and P. Jackson The functional architecture of human empathy. Behavioral and Cognitive Neuroscience Reviews 3 (2), p. 71ā100. Cited by: Appendix J, Ā§1. Demir et al. (2026) A. Demir, H. Aydın, K. A. Tessera, D. Abel, and S. Albrecht Fairness over equality: correcting social incentives in asymmetric sequential social dilemmas. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), Cited by: Appendix J. Dong et al. (2024) S. Dong, C. Li, S. Yang, B. An, W. Li, and Y. Gao Egoism, utilitarianism and egalitarianism in multi-agent reinforcement learning. Neural Networks 178, p. 106544. Cited by: Appendix J. Eccles et al. (2019) T. Eccles, E. Hughes, J. KramĆ”r, S. Wheelwright, and J. Leibo Learning reciprocity in complex sequential social dilemmas. External Links: 1903.08082 Cited by: Appendix J. Epley et al. (2004) N. Epley, B. Keysar, L. Van Boven, and T. Gilovich Perspective taking as egocentric anchoring and adjustment. Journal of Personality and Social Psychology 87 (3), p. 327ā339. Cited by: Appendix J, Ā§5.2.5. Fehr and GƤchter (2000) E. Fehr and S. GƤchter Cooperation and punishment in public goods experiments. American Economic Review 90 (4), p. 980ā994. Cited by: Appendix J, Ā§1. Fehr and Schmidt (1999) E. Fehr and K. Schmidt A theory of fairness, competition, and cooperation. The Quarterly Journal of Economics 114 (3), p. 817ā868. Cited by: Appendix J, Appendix J, Ā§1, Ā§2.3, Ā§2.3. Foerster et al. (2018a) J. Foerster, R. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch Learning with opponent-learning awareness. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), p. 122ā130. Cited by: Appendix J, Table 13. Foerster et al. (2018b) J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence, p. 2974ā2982. Cited by: Appendix J, Table 13, Ā§1, Ā§3. Gemp et al. (2022) I. Gemp, K. McKee, R. Everett, E. DuƩƱez-GuzmĆ”n, Y. Bachrach, D. Balduzzi, and A. Tacchetti D3C: reducing the price of anarchy in multi-agent learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), p. 498ā506. Cited by: Appendix J. Goldman (2006) A. Goldman Simulating minds: the philosophy, psychology, and neuroscience of mindreading. Oxford University Press. Cited by: Appendix J, Ā§1. Gordon (1986) R. Gordon Folk psychology as simulation. Mind & Language 1 (2), p. 158ā171. Cited by: Appendix J, Ā§1, Ā§6. Grupen et al. (2022) N. Grupen, B. Selman, and D. Lee Cooperative multi-agent fairness and equivariant policies. In AAAI Conference on Artificial Intelligence, p. 9350ā9359. Cited by: Appendix J. Hadfield-Menell et al. (2016) D. Hadfield-Menell, A. Dragan, P. Abbeel, and S. Russell Cooperative inverse reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), p. 3909ā3917. Cited by: Appendix J. Haupt et al. (2024) A. Haupt, P. Christoffersen, M. Damani, and D. Hadfield-Menell Formal contracts mitigate social dilemmas in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems 38 (51). Cited by: Appendix J. He et al. (2016) H. He, J. Boyd-Graber, K. Kwok, and H. DaumĆ© I Opponent modeling in deep reinforcement learning. In International Conference on Machine Learning (ICML), p. 1804ā1813. Cited by: Appendix J. Henrich et al. (2001) J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr, H. Gintis, and R. McElreath In search of homo economicus: behavioral experiments in 15 small-scale societies. American Economic Review 91 (2), p. 73ā78. Cited by: Appendix J, Ā§1. Hughes et al. (2018) E. Hughes, J. Leibo, M. Phillips, K. Tuyls, E. DuƩƱez-GuzmĆ”n, A. GarcĆa CastaƱeda, I. Dunning, T. Zhu, K. McKee, R. Koster, H. Roff, and T. Graepel Inequity aversion improves cooperation in intertemporal social dilemmas. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Appendix J, Table 13, Ā§1, Ā§2.3, Ā§2.3, Ā§2.3, Ā§4.3, Ā§5.1.1, Ā§5.1.2, Ā§5.2.1, Ā§6. Jaques et al. (2019) N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, P. A. Ortega, D. Strouse, J. Z. Leibo, and N. de Freitas Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning (ICML), p. 3040ā3049. Cited by: Appendix J, Table 13, Ā§3, Ā§6. Jara-Ettinger (2019) J. Jara-Ettinger Theory of mind as inverse reinforcement learning. Current Opinion in Behavioral Sciences 29, p. 105ā110. Cited by: Appendix J, Appendix J. Jiang and Lu (2019) J. Jiang and Z. Lu Learning fairness in multi-agent systems. In Advances in Neural Information Processing Systems (NeurIPS), p. 13854ā13865. Cited by: Appendix J. Jin et al. (2025) Y. Jin, S. Wei, and G. Montana Achieving collective welfare in multi-agent reinforcement learning via suggestion sharing. Machine Learning 114 (8), p. 190. Cited by: Appendix J, Ā§6. Kaneko and Nakamura (1979) M. Kaneko and K. Nakamura The Nash social welfare function. Econometrica 47 (2), p. 423ā435. Cited by: Appendix J, Ā§5.1.3. Kleiman-Weiner et al. (2016) M. Kleiman-Weiner, M. Ho, J. Austerweil, M. Littman, and J. Tenenbaum Coordinate to cooperate or compete: abstract goals and joint intentions in social interaction. In Annual Meeting of the Cognitive Science Society (CogSci), Cited by: Appendix J. Kong et al. (2024) F. Kong, Y. Huang, S. Zhu, S. Qi, and X. Feng Learning to balance altruism and self-interest based on empathy in mixed-motive games. In Advances in Neural Information Processing Systems (NeurIPS), p. 135819ā135842. Cited by: Appendix J, Ā§6. Kƶster et al. (2020) R. Kƶster, K. McKee, R. Everett, L. Weidinger, W. Isaac, E. Hughes, E. DuƩƱez-GuzmĆ”n, T. Graepel, M. Botvinick, and J. Leibo Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences. External Links: 2010.09054 Cited by: Appendix J. Leibo et al. (2021) J. Z. Leibo, E. A. DuƩƱez-GuzmĆ”n, A. S. Vezhnevets, J. Agapiou, P. Sunehag, R. Koster, J. Matyas, C. Beattie, I. Mordatch, and T. Graepel Scalable evaluation of multi-agent reinforcement learning with Melting Pot. In International Conference on Machine Learning (ICML), p. 6187ā6199. Cited by: Appendix J, Ā§2.3. Leibo et al. (2017) J. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel Multi-agent reinforcement learning in sequential social dilemmas. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), p. 464ā473. Cited by: Appendix J, Ā§1, Ā§2.1. Lerer and Peysakhovich (2018) A. Lerer and A. Peysakhovich Maintaining cooperation in complex social dilemmas using deep reinforcement learning. External Links: 1707.01068 Cited by: Appendix J. Lowe et al. (2017) R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems (NeurIPS), p. 6382ā6393. Cited by: Appendix J, Table 13, Ā§1, Ā§3. Lupu and Precup (2020) A. Lupu and D. Precup Gifting in multi-agent reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), p. 789ā797. Cited by: Appendix J, Ā§6. McKee et al. (2020) K. McKee, I. Gemp, B. McWilliams, E. DuƩƱez-GuzmĆ”n, E. Hughes, and J. Leibo Social diversity and social preferences in mixed-motive reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), p. 869ā877. Cited by: Appendix J, Table 13, Ā§1, Ā§2.3, Ā§2.3, Ā§6. Messick and McClintock (1968) D. Messick and C. McClintock Motivational bases of choice in experimental games. Journal of Experimental Social Psychology 4 (1), p. 1ā25. Cited by: Ā§2.3, Ā§2.3. Natarajan et al. (2010) S. Natarajan, G. Kunapuli, K. Judah, P. Tadepalli, K. Kersting, and J. Shavlik Multi-agent inverse reinforcement learning. In International Conference on Machine Learning and Applications (ICMLA), p. 395ā400. Cited by: Appendix J, Ā§6. Ng et al. (1999) A. Ng, D. Harada, and S. Russell Policy invariance under reward transformations: theory and application to reward shaping. In International Conference on Machine Learning (ICML), p. 278ā287. Cited by: Appendix J. Ng and Russell (2000) A. Y. Ng and S. J. Russell Algorithms for inverse reinforcement learning. In International Conference on Machine Learning (ICML), p. 663ā670. Cited by: Appendix J, Ā§6. Perolat et al. (2017) J. Perolat, J. Z. Leibo, V. Zambaldi, C. Beattie, K. Tuyls, and T. Graepel A multi-agent reinforcement learning model of common-pool resource appropriation. In Advances in Neural Information Processing Systems (NeurIPS), p. 3646ā3655. Cited by: Appendix J, Ā§1, Ā§2.1, Ā§5.1.1, Ā§5.1.3. Peysakhovich and Lerer (2018) A. Peysakhovich and A. Lerer Prosocial learning agents solve generalized stag hunts better than selfish ones. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), p. 2043ā2044. Cited by: Appendix J, Table 13, Ā§1, Ā§2.3, Ā§6. Preston and de Waal (2002) S. Preston and F. de Waal Empathy: its ultimate and proximate bases. Behavioral and Brain Sciences 25 (1), p. 1ā20. Cited by: Appendix J, Ā§1, Ā§6. Rabinowitz et al. (2018) N. Rabinowitz, F. Perbet, F. Song, C. Zhang, S. M. A. Eslami, and M. Botvinick Machine theory of mind. In Proceedings of the 35th International Conference on Machine Learning (ICML), p. 4218ā4227. Cited by: Appendix J, Ā§3. Raileanu et al. (2018) R. Raileanu, E. Denton, A. Szlam, and R. Fergus Modeling others using oneself in multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), p. 4257ā4266. Cited by: Appendix J, Ā§3, Ā§6. Rashid et al. (2018) T. Rashid, M. Samvelyan, C. Schroeder de Witt, G. Farquhar, J. Foerster, and S. Whiteson QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), p. 4295ā4304. Cited by: Appendix J, Table 13. Rawls (1971) J. Rawls A theory of justice. Harvard University Press. Cited by: Appendix J. Ross et al. (1977) L. Ross, D. Greene, and P. House The āfalse consensus effectā: an egocentric bias in social perception and attribution processes. Journal of Experimental Social Psychology 13 (3), p. 279ā301. Cited by: Appendix J, Ā§5.2.5. Schulman et al. (2016) J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations (ICLR), Cited by: Ā§A.1, Ā§2.2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. External Links: 1707.06347 Cited by: Ā§2.2. Senadeera et al. (2024) M. Senadeera, T. Karimpanal George, S. Jacobs, S. Gupta, and S. Rana EMOTE: an explainable architecture for modelling the other through empathy. In International Joint Conference on Artificial Intelligence, IJCAI, p. 4876ā4884. Cited by: Appendix J. Senadeera et al. (2022) M. Senadeera, T. G. Karimpanal, S. Gupta, and S. Rana Sympathy-based reinforcement learning agents. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), p. 1164ā1172. Cited by: Appendix J. Siddique et al. (2020) U. Siddique, P. Weng, and M. Zimmer Learning fair policies in multiobjective (deep) reinforcement learning with average and discounted rewards. In International Conference on Machine Learning (ICML), p. 8905ā8915. Cited by: Appendix J. Sunehag et al. (2018) P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, and T. Graepel Value-decomposition networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), p. 2085ā2087. Cited by: Appendix J, Table 13. Sutton and Barto (2018) R. Sutton and A. Barto Reinforcement learning: an introduction. 2nd edition, MIT Press. Cited by: Ā§A.1, Ā§2.2. Sutton et al. (1999) R. Sutton, D. McAllester, S. Singh, and Y. Mansour Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NeurIPS), p. 1057ā1063. Cited by: Ā§A.1, Ā§2.2. Tennant et al. (2023) E. Tennant, S. Hailes, and M. Musolesi Modeling moral choices in social dilemmas with multi-agent reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI), p. 317ā325. Cited by: Appendix J. Vinitsky et al. (2023) E. Vinitsky, R. Kƶster, J. Agapiou, E. DuƩƱez-GuzmĆ”n, A. Vezhnevets, and J. Leibo A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings. Collective Intelligence 2 (2). Cited by: Appendix J. Wang et al. (2019) J. X. Wang, E. Hughes, C. Fernando, W. Czarnecki, E. DuƩƱez-GuzmĆ”n, and J. Leibo Evolving intrinsic motivations for altruistic behavior. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), p. 683ā692. Cited by: Appendix J. Wei et al. (2026) Y. Wei, Y. Ogura, Y. Ohmura, I. M. de Abril, H. Kanazawa, and Y. Kuniyoshi Integrated altruistic and fairness preference induces advanced mutual cooperation in sequential social dilemmas. External Links: 2607.04710 Cited by: Appendix J, Ā§6. Williams (1992) R. Williams Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 (3), p. 229ā256. Cited by: Ā§A.1, Ā§2.2. Willis et al. (2024) R. Willis, Y. Du, J. Leibo, and M. Luck Resolving social dilemmas with minimal reward transfer. Autonomous Agents and Multi-Agent Systems 38 (2). Cited by: Appendix J, Ā§6. Wolpert and Tumer (2001) D. Wolpert and K. Tumer Optimal payoff functions for members of collectives. Advances in Complex Systems 4 (2ā3), p. 265ā279. Cited by: Appendix J. Yang et al. (2020) J. Yang, A. Li, M. Farajtabar, P. Sunehag, E. Hughes, and H. Zha Learning to incentivize other learning agents. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Appendix J, Ā§1, Ā§5.1.1, Ā§6. Yi et al. (2022) Y. Yi, G. Li, Y. Wang, and Z. Lu Learning to share in networked multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), p. 15119ā15131. Cited by: Appendix J. Yoshida and Man (2025) N. Yoshida and K. Man Homeostatic coupling for prosocial behavior. External Links: 2506.12894 Cited by: Appendix J. Yu et al. (2022) C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu The surprising effectiveness of PPO in cooperative multi-agent games. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, p. 24611ā24624. Cited by: Appendix J. Yu et al. (2019) L. Yu, J. Song, and S. Ermon Multi-agent adversarial inverse reinforcement learning. In International Conference on Machine Learning (ICML), p. 7194ā7201. Cited by: Appendix J. Zimmer et al. (2021) M. Zimmer, C. Glanois, U. Siddique, and P. Weng Learning fair policies in decentralized cooperative multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), p. 12967ā12978. Cited by: Appendix J. These appendices give the details and the evidence behind the paper. Appendices A and B describe the implementation and the experimental protocol. Appendices C to H extend the subsections of results, one each and in the same order. Appendix I collects the limitations of the study, and Appendix J reviews the related work in full. Figures, tables, and equations numbered with an S belong to the appendices. Appendix A Method Details This appendix elaborates on the methods section. Specifically we discuss why we can use the social terms within the coefficient of the policy update, the order of one training iteration, how the reward model is trained, how the scores are scaled, and how episode boundaries are handled. A.1. Policy Gradients and the Baseline The background section uses two facts about policy gradients: the advantage measures how much better an action went than expected, and a term that does not depend on the action leaves the expected gradient unchanged. Both follow from the policy gradient theorem (Sutton et al., 1999; Williams, 1992), (16) āJā(Īøi)=ā”[ātālogā”Ļiā(aitā£oit)ā(Gitābā”(oit))],ā J( _i)=E [ _tā _i(a_i^t o_i^t)\, (G_i^t-b(o_i^t) ) ], where Git=ākā„0Ī³kārit+kG_i^t= _kā„ 0Ī³^kr_i^t+k is the return that followed action aita_i^t and b is a baseline, any function that does not depend on the action. The baseline does not change the gradient in expectation, because (17) āab(o)āĻi(aā£o)=b(o)āāaĻi(aā£o)=b(o)ā1=0, _ab(o)\,ā _i(a o)=b(o)\,ā _a _i(a o)=b(o)\,ā 1=0, but it reduces the variance of the estimate (Sutton and Barto, 2018). Actor-critic methods use a learned critic Viā(o)āā”[Gitā£oit=o]V_i(o) [G_i^t o_i^t=o] as the baseline and estimate the advantage GitāViā(oit)G_i^t-V_i(o_i^t) by bootstrapping. With the temporal-difference error, the generalized advantage estimate (Schulman et al., 2016) is Ī“it _i^t =rit+Ī³āViā(oit+1)āViā(oit), =r_i^t+Ī³ V_i(o_i^t+1)-V_i(o_i^t), (18) Ait A_i^t =ākā„0(Ī³āĪ»)kāĪ“it+k, = _kā„ 0(Ī³Ī»)^k _i^t+k, where Ī»ā[0,1]Ī»ā[0,1] trades bias for variance; with Ī»=1Ī»=1, Ait=GitāViā(oit)A_i^t=G_i^t-V_i(o_i^t). Eq. (17) is why the coefficient of the policy update is a separate place for a social term: a term added to AitA_i^t changes the expected update only through its dependence on the action. It is also why SRA is not simply a reweighted version of SRR. The cross scores it adds depend on another agentās action, not on iās, so they do not vanish in expectation, but they also do not correspond to any fixed objective that i maximizes. A.2. One Training Iteration Table 1 summarizes what each learner optimizes. The four rows differ in two columns only, the reward that PPO and the critic see and the coefficient that weights the actorās update, which is the distinction the paper turns on: SRR changes the first and SRA the second. Algorithm 1 gives one training iteration of agent i. The order matters in three places. The rollout is collected with the networks and reward models frozen, so every self-referenced estimate within an iteration comes from the same model. The SRR term and the standing gate are applied in the order of the transitions, with the standing from before the current episode, so an agent is never penalized for a lead it acquires in the episode being scored. And all networks are updated only after the social coefficients of the rollout are fixed, so a reward model is used from the rollout after the one it was fitted on. Evaluating N reward models and critics on N streams costs ā”(N2)O(N^2) forward passes per step, which is the main cost of the method over independent PPO. Table 1. The learning signal of each learner. ViobjV_i obj is trained on the return of the reward in the second column. Learner PPO reward actor coefficient fif_i target Plain PPO rir_i own GAE ā TR ri+Fiā(xtrue)r_i+F_i(x true) own GAE ā SRR ri+Fiā(xpred)r_i+F_i(x pred) own GAE rir_i SRA rir_i ĪØ~i _i rir_i + Standing above āĪŗāaheadiā[ri]+-Īŗ\,ahead_i[r_i]_+ unchanged unchanged Algorithm 1 One training iteration for agent i. 1: Collect all transition streams; reveal only rir_i to learner i 2: Evaluate r^iājtāfiā(ojt,ajt,ojt+1) r_ij^tā f_i(o_j^t,a_j^t,o_j^t+1); set r^iāitārit r_i^tā r_i^t 3: if reward-model warm-up is complete then 4: For SRR, compute xiājx_ij and uiāri+Fi(xiā )u_iā r_i+F_i(x_iĀ·) 5: Apply standing adjustment using the pre-update ledger; then update GiG_i with completed episodes 6: else 7: uiāriu_iā r_i 8: end if 9: Compute own GAE from uiu_i; for SRA compute A^iāj A_ij and ĪØ~i _i 10: Optimize actor and objective critic; regress fif_i on own current and replay transitions A.3. Reward-Model Training Each reward model is trained on the current minibatch and, once its replay holds any transitions, on an independently sampled replay minibatch, (19) āiRM=12ā¬i(fiāri)2+[|āi|>0]12āi(fiāri)2.L^RM_i= 12E_B_i(f_i-r_i)^2+1[|R_i|>0]\, 12E_R_i(f_i-r_i)^2. The replay belongs to agent i alone, holds only its own transitions with a nonzero reward, up to 2,048 of them, and is sampled with replacement. Its purpose is coverage rather than efficiency: an agent that settles into cleaning the river stops eating apples, and without the replay its minibatches would soon contain no example of an apple reward, so it would forget how to recognize one in another agentās transition. The estimates that enter the social terms are computed before the update and carry no gradient into the policy, so the reward model is fitted only to the agentās own rewards and never to make the social term convenient. The social term stays off for the first W environment steps, 20k in the three environments, while the model is fitted. With value clipping, the critic loss is (20) āVi=12max(ViāRi)2,[Viold+clip(ViāViold,āĻµc,Ļµc)āRi]2, splitL_V_i= 12E \&(V_i-R_i)^2,\\ &[V_i^old+clip(V_i-V_i^old,- _c, _c)-R_i]^2\, split and each agent minimizes (21) āi=āiactor+cVāāViācHā[āā”(Ļi)]+rawācVāāViraw+SRācfāāiRM, splitL_i=&L^actor_i+c_VL_V_i-c_HE[H( _i)]\\ &+1_rawc_VL_V_i^raw+1_SRc_fL^RM_i, split where āiactorL^actor_i is PPOās clipped objective with the coefficient of Table 1 in place of the advantage, the raw critic VirawV_i^raw appears only in the control of Appendix F.3, and the reward-model term only for self-referenced learners. The actor and the critic of an agent share its observation encoder, no parameters are shared between agents, and gradients are clipped for each agent separately. A.4. Scaling the Scores In SRA, the agentās own score and each cross score are standardized over the time steps and parallel environments of a rollout before they are combined, (22) rollā(x)=xāĪ¼t,eā(x)maxā”(Ļt,eā(x),10ā6),S_roll(x)= x- _t,e(x) ( _t,e(x),10^-6), so a constant score becomes zero, and PPO normalizes the combined coefficient once more within each minibatch. Standardizing first is what makes a single weight Ī± meaningful across games: the cross scores of an agent in Commons Harvest, where a beam costs 50, would otherwise dwarf those in Escape Room, where a lever costs 1. The price is that Ī± and Ī² in SRA weight standardized scores rather than rewards, so their values are not comparable with those of SRR or TR, and a larger weight in one is not a stronger preference in the other. The social terms average over the other agents, dividing by Nā1N-1; a variant that takes the largest gap instead of the mean is not used in any reported result. A.5. Episode Boundaries An episode ends either because the agents finish or because of a time limit, and the two need different treatment. At a true end there is nothing to bootstrap, so the cross score defined in the main paper uses no next value and its recursion stops. At a time limit the episode could have continued, so the score keeps the value of the last observation, but the recursion still stops, so that one episode is never linked to the next. The look-ahead of SRR+V is dropped only at true ends, where there is no future to anticipate. For the agentās own advantage we add the value of the last observation to the final reward at a time limit, which gives the same result as bootstrapping it. Getting this wrong matters most in Escape Room, where episodes are five steps long, so a score that leaked across boundaries would mix the outcome of one attempt into the next. Appendix B Experimental Details This appendix describes what the agents observe and can do, how they are built and trained, how every reported setting was chosen, and how the partial-view experiments differ from the main ones. B.1. Observations and Actions Table 2 lists what each agent observes and can do, and Figure 5 shows one state of each game as two different agents see it, side by side, with the full view in the top row and the partial view below. The figure makes concrete the property the method depends on: every observation is arranged around its own agent, so the same reward model and critic can be applied to any agentās observation. In Escape Room the observation is a stack of one-hot rows, the observerās own place first and the others below in a fixed order; agents 1 and 4 therefore see the same state as two different matrices. In Clean Up the window is centred on the observer, drawn here with one colour per plane, and in Commons Harvest it is centred and also turned to the direction the agent faces, so the two columns show the same map at different angles. The bottom row shows what the partial view removes. With the full view the window is large enough to contain the whole map from any position, so agent i sees in ojo_j everything agent j sees, and assessing jās transition is a matter of recognizing the event. With the 15Ć1515Ć 15 window, much of ojo_j lies outside iās own window: in the example, agent 1ās Clean Up window contains the river bank and two other agents, while agent 2ās contains mostly river and waste. Agent i must then judge jās situation from a view it has never had itself, which is the difficulty that Appendix G examines. Escape Room has no partial version, because its observation is already the list of places rather than a map. Figure 5. One state of each game as two agents observe it, after a few random steps, with the full view (top) and the partial view (bottom). Escape Room (N=5N=5): the one-hot place of every agent, the observerās own first and the others in a fixed order; agent 1 is at the lever and agent 4 at the door. The observation is not spatial, so it has no partial version. Clean Up: the 47Ć4747Ć 47 window centred on the observer (red circle), large enough to contain the whole 25Ć1825Ć 18 map from any position, shown with one colour per plane (orange self, blue others, green apples, brown waste, light blue river, grey walls; beams yellow). Commons Harvest: the 73Ć7373Ć 73 RGB window centred on the observer and turned to the direction it faces, so the two agents see the map at different angles; apples are green, the observer and the others have one fixed colour each. The partial view is the central 15Ć1515Ć 15 part of the same window (dashed box, top), used in the partial-view experiments of Appendix G. Everything outside the map is empty. Table 2. Observation and actions of one agent. Every observation is arranged relative to its agent. The full view covers the whole state and is used in the main experiments; the partial view keeps only the 15Ć1515Ć 15 neighbourhood of the agent. Figure 5 shows examples of both. Game Observation Actions Escape Room one-hot place (start, lever, door) of every agent, its own first; 3āN3N values; no partial view 3: stay, lever, door Clean Up window centred on the agent, 7 binary planes: self, others, apple, waste, river, wall, beam; 47Ć4747Ć 47 full, 15Ć1515Ć 15 partial 6: four moves, stay, clean Commons Harvest RGB window centred on the agent and turned to its facing direction, self and others each in one fixed colour; 73Ć7373Ć 73 full, 15Ć1515Ć 15 partial 8: four moves, stay, two turns, fire B.2. Networks and PPO Settings Table 3 lists the networks and PPO settings of every game, and Table 4 the PPO setting of each Escape Room learner with five agents. All learners use Ī³=0.99Ī³=0.99, GAE Ī»=0.95Ī»=0.95, clip 0.20.2, four epochs of four minibatches per update, value coefficient 0.50.5, gradient-norm clip 0.50.5, and a linearly annealed learning rate. The reward model is the one component we had to adapt per game. In Escape Room and Clean Up it is a separate network on the transition, small enough to fit quickly. In Commons Harvest, where the observation is a 73Ć7373Ć 73 image, a separate convolutional network was slow to fit from the few rewarded transitions available, so the model is instead a small head on the policy encoderās features of o and oā²o . The features are detached, so fitting the reward model does not change the encoder, and the policy is unaffected by the reward model except through the social term. Table 3. Networks and PPO settings. The actor and the critic are linear heads on one shared network. Escape Room tunes the PPO setting per learner (Table 4); Clean Up and Commons Harvest use one setting for all learners, chosen in the PPO stage of the selection protocol. Escape Room Clean Up Commons Harvest Shared network MLP, two layers of 64 tanh units 3Ć33Ć 3 convolutions with 16 and 32 channels (second with stride 2), 128 ReLU units CleanRL Atari network: convolutions 8Ć88Ć 8/4 (32), 4Ć44Ć 4/2 (64), 3Ć33Ć 3/1 (64), 512 ReLU units Reward model fif_i MLP, two layers of 64 tanh units, on (o,a,oā²)(o,a,o ) two 3Ć33Ć 3 convolutions (16 channels) on the stacked (o,oā²)(o,o ), max- and mean-pooled, then 64 units with a one layer of 64 tanh units on the shared networkās features of o and oā²o (not trained through) and a Steps; parallel envs 500k; 16 5M; 8 5M; 8 Rollout per env 16, 32, or 128 (tuned) 256 128 Learning rate 2.5Ć10ā42.5Ć 10^-4 or 10ā310^-3 (tuned) 10ā310^-3 10ā310^-3 Entropy coefficient 0.01 or 0.05 (tuned) 0.01 (N=5N=5); 0.05 (N=3N=3)ā 0.01 PPO stage joint with the operator weights learning rate 2.5Ć10ā4,10ā3\2.5Ć 10^-4,10^-3\ Ć entropy 0.01,0.05\0.01,0.05\ Ć rollout 128,256,512\128,256,512\ on plain PPO and six social configurations, three seeds learning rate 2.5Ć10ā4,10ā3\2.5Ć 10^-4,10^-3\ Ć entropy 0.003,0.01,0.03\0.003,0.01,0.03\ on plain PPO, two seeds Warm-up W 20k steps 20k steps 20k steps Tuning; held-out seeds 1ā3; 11ā18 1ā3; 7ā14 1ā3; 4ā11 āAt N=3N=3 plain PPO uses its own tuned setting: learning rate 2.5Ć10ā42.5Ć 10^-4, entropy 0.01, rollout 128. Table 4. Escape Room, N=5N=5: PPO setting of each learner at its collective-return pick (learning rate, entropy coefficient, rollout length per environment). Learner (weights) LR, entropy Rollout Plain PPO 10ā310^-3, 0.01 32 TR-EI (Ī±=1Ī±=1) 10ā310^-3, 0.01 128 TR-IA (Ī±/Ī²=0/1Ī±/Ī²=0/1) 10ā310^-3, 0.01 32 TR-SVO (Ī±=1Ī±=1, Ļ=Ļ/4Ļ=Ļ/4) 10ā310^-3, 0.01 128 SRA-EI (Ī±=3Ī±=3) 2.5Ć10ā42.5Ć 10^-4, 0.05 128 SRA-IA (Ī±/Ī²=0/0.5Ī±/Ī²=0/0.5) 10ā310^-3, 0.05 128 SRA-SVO (Ī±=5Ī±=5, Ļ=Ļ/3Ļ=Ļ/3) 2.5Ć10ā42.5Ć 10^-4, 0.01 128 SRR-EI (Ī±=0.1Ī±=0.1) 10ā310^-3, 0.05 128 SRR-IA (Ī±/Ī²=0/1Ī±/Ī²=0/1) 2.5Ć10ā42.5Ć 10^-4, 0.05 128 SRR-IA+V (Ī±/Ī²=0/1Ī±/Ī²=0/1) 2.5Ć10ā42.5Ć 10^-4, 0.01 128 SRR-IA + Standing (Īŗ=1Īŗ=1) 2.5Ć10ā42.5Ć 10^-4, 0.05 128 B.3. Hyperparameter Selection Every number in the paper comes from settings chosen before the held-out runs. In Clean Up and Commons Harvest we first choose one PPO setting and keep it for every learner, so that no learner is favoured by a better base configuration; in Escape Room, where runs are cheap, we tune the PPO setting together with each learnerās social weights. We then score every candidate by the mean minus the sample standard deviation of the metric across the tuning seeds, once with collective return and once with Nash welfare, on seeds 1ā3 in every game. Subtracting one standard deviation favours settings that work on every tuning seed rather than settings with one good seed; with two or three seeds it is a rule of thumb, not a confidence bound. Each chosen setting is then trained from scratch on eight held-out seeds, 11ā18 in Escape Room, 7ā14 in Clean Up, and 4ā11 in Commons Harvest, and nothing from these runs feeds back into the choice. When both metrics choose the same setting, it is run once. Figure 6 shows why we select twice. Each point is one Clean Up setting, placed by its mean collective return and mean Nash welfare over the tuning seeds, with one panel per group size and one colour per learner family. For most of the grid the two metrics agree: the cloud rises along a diagonal, where a setting that produces more also divides it more evenly, simply because the alternative to producing is that nobody eats. The cloud then splits. Above about 800 apples with three agents and 1,000 with five, one branch keeps rising and the other turns right and down, toward settings that produce more but leave some agents with little. The colours show who is on each branch. The branch that turns down holds the true-reward learners, in orange, and many reward-placement settings, in green; the branch that keeps rising is mostly SRA, in blue, joined by the best SRR+V settings. At every group size the setting selected on collective return, the circle, is a true-reward learner at the far right with a Nash welfare near the bottom of the range, and the setting selected on Nash welfare, the diamond, is a self-referenced learner at the top: SRA-EI with three and seven agents and SRR-IA+V with five. No setting is at the corner of both, which is why we report both selections throughout rather than choosing one. The grids hold 132 candidates in Clean Up with three agents, 108 with five, 132 with seven, and 40 in Commons Harvest, including the self-scaling controls; in Escape Room they cover every operator, placement, and standing setting. Within a learner, the two metrics choose different settings for 7 of 21 learners with three agents, 5 of 21 with five, and 6 of 21 with seven, and in Escape Room for 15 of 126 combinations of group size and learner. Where they differ, the setting chosen on Nash welfare often has a slightly lower mean but varies much less across seeds. Figure 6. Clean Up tuning landscape. Each point is one setting evaluated on the three tuning seeds, placed by its mean collective return and mean Nash welfare and coloured by learner family; grey points are variants not analysed in the paper. The circle and the diamond mark the settings that the mean-minus-standard-deviation score selects on collective return and on Nash welfare, among all candidates at that group size. B.4. Partial View For the partial view, every learner is retuned and trained for 7 million environment steps, more than the 5 million of the full view, because the smaller window makes the task slower to learn for every learner including plain PPO. The grid scales each learnerās full-view social weights by 13 13, 1, and 3 and crosses them with learning rates 10ā310^-3 and 3Ć10ā43Ć 10^-4 and entropy coefficients 0.01 and 0.05, on tuning seeds 1 and 2; the selected setting is rerun on the same held-out seeds as with the full view. Scaling the full-view weights rather than searching from scratch keeps the grid small, and it assumes that the right weight moves with the view but not far, which the selected settings bear out. The recurrent agents carry a GRU (Cho et al., 2014) in the policy and in the critic over the agentās observation, its previous action, and its previous reward, with layer normalization, a skip connection around the recurrent layer, and truncated backpropagation over 64 steps. The reward model stays feed-forward and reads only the transition, so memory can help the agent act and estimate values but is not available to the self-referenced estimate itself. Appendix C Can Agents Cooperate without Observing Othersā Rewards? Can social preferences be grounded in self-referenced estimates of othersā outcomes rather than in their reward signals? This appendix gathers the evidence. It places our learners among all the learners we ran, gives the full results, adds the two measures that collective return leaves out, and asks how far the Escape Room result extends. Two conclusions emerge. Self-referenced estimates are enough to start and sustain cooperation in all three games. Where they fall short of the true rewards, in Commons Harvest, the extra production of the true-reward learners goes to a few agents. C.1. Where Our Learners Stand Figure 7 places our three self-referenced learners among all the learners we ran, on what each game tests: cooperation in Escape Room, productivity and fairness in Clean Up, and efficiency, sustainability, and welfare in Commons Harvest. Each axis spans the means of all learners in that game, true-reward learners included, so reaching the edge means doing as well as the best learner we ran, not merely better than plain PPO. Plain PPO is the small region in the middle; it reaches the welfare axis only because agents that each earn almost nothing are equal. Our learners extend it in different directions, SRA-EI towards production and SRR-IA+V towards fairness, and between them they reach the edge on most axes. The two axes where the true-reward learners still lead, efficiency and sustainability in Commons Harvest, are the shortfall that the next figures trace to how the return is divided. Radar plot comparing plain PPO with three self-referenced learners on six properties across the three games. Figure 7. Our three self-referenced learners, which never observe another agentās reward, against plain PPO on what each game tests: cooperation in Escape Room, productivity and fairness in Clean Up, and efficiency, sustainability, and welfare in Commons Harvest. Each axis spans the means of all learners in that game, true-reward ones included; in Commons Harvest the true-reward agents reach further on efficiency and sustainability. Eight held-out seeds.Radar plot comparing plain PPO with three self-referenced learners on six properties across the three games. C.2. The Full Results Table 5 gives the final collective return, Nash welfare, and lowest agent return of every learner in every game, at both of its selections. Table 6 adds sustainability for Commons Harvest, where a high collective return can also come from stripping the map early, and Table 7 adds a behavioural measure for Escape Room and Clean Up with five agents: how spread the exit rates are, and how much of the cleaning the two poorest agents do. Two entries in Table 6 change how the Commons Harvest results should be read. TR-IA ends with a collective return of ā1,935-1,935, which only the beam can produce, so its agents are not merely failing to harvest but firing at one another. And the self-scaling control, which rescales the agentās own reward and uses no information about the others, has the highest Nash welfare and the highest lowest agent return of any learner in the game. A high welfare in this game can therefore be reached without any preference over the others, which is why we report the control separately and do not read welfare there as evidence of cooperation. Table 5. Every learner at the hyperparameter setting selected on tuning seeds by collective return and by Nash welfare (āsame settingā when both objectives selected it). C, āNW, and L are computed from each seedās final-window per-agent means. Entries are held-out-seed means Ā± 95% Student-t intervals; they describe the frozen selections and are not used to reselect them. selected on collective return selected on Nash welfare Learner C āNW L C āNW L Escape Room (N=5N=5) Plain PPO ā0.3Ā±0.4-0.3Ā± 0.4 4.95Ā±0.084.95Ā± 0.08 ā0.25Ā±0.39-0.25Ā± 0.39 same setting TR-EI 17.0Ā±0.017.0Ā± 0.0 6.96Ā±0.006.96Ā± 0.00 ā1.00Ā±0.00-1.00Ā± 0.00 same setting SRA-EI 15.7Ā±3.015.7Ā± 3.0 6.78Ā±0.426.78Ā± 0.42 ā1.00Ā±0.00-1.00Ā± 0.00 same setting SRR-EI 0.0Ā±0.00.0Ā± 0.0 5.00Ā±0.005.00Ā± 0.00 0.00Ā±0.000.00Ā± 0.00 same setting TR-IA 17.0Ā±0.017.0Ā± 0.0 6.96Ā±0.006.96Ā± 0.00 ā1.00Ā±0.00-1.00Ā± 0.00 17.0Ā±0.017.0Ā± 0.0 6.96Ā±0.006.96Ā± 0.00 ā1.00Ā±0.00-1.00Ā± 0.00 SRA-IA 0.0Ā±0.00.0Ā± 0.0 5.00Ā±0.005.00Ā± 0.00 0.00Ā±0.000.00Ā± 0.00 same setting SRR-IA+V 17.0Ā±0.017.0Ā± 0.0 6.96Ā±0.006.96Ā± 0.00 ā1.00Ā±0.00-1.00Ā± 0.00 same setting TR-SVO 14.9Ā±5.014.9Ā± 5.0 6.72Ā±0.586.72Ā± 0.58 ā0.88Ā±0.30-0.88Ā± 0.30 same setting SRA-SVO 4.7Ā±8.24.7Ā± 8.2 5.41Ā±1.145.41Ā± 1.14 ā1.00Ā±0.00-1.00Ā± 0.00 same setting SRR-SVO 0.0Ā±0.00.0Ā± 0.0 5.00Ā±0.005.00Ā± 0.00 0.00Ā±0.000.00Ā± 0.00 same setting Clean Up (N=5N=5) Plain PPO 248Ā±198248Ā± 198 37.0Ā±29.837.0Ā± 29.8 5.4Ā±4.85.4Ā± 4.8 same setting TR-EI 1,530Ā±2891,530Ā± 289 70.4Ā±33.670.4Ā± 33.6 0.0Ā±0.00.0Ā± 0.0 1,159Ā±4441,159Ā± 444 86.7Ā±14.686.7Ā± 14.6 3.3Ā±5.23.3Ā± 5.2 SRA-EI 1,286Ā±1411,286Ā± 141 214.4Ā±15.3214.4Ā± 15.3 83.1Ā±16.683.1Ā± 16.6 same setting SRR-EI 1,512Ā±1401,512Ā± 140 84.5Ā±29.884.5Ā± 29.8 0.0Ā±0.00.0Ā± 0.0 same setting TR-IA 964Ā±107964Ā± 107 178.1Ā±20.6178.1Ā± 20.6 95.9Ā±18.095.9Ā± 18.0 same setting SRA-IA 590Ā±188590Ā± 188 101.6Ā±38.5101.6Ā± 38.5 39.5Ā±32.839.5Ā± 32.8 same setting SRR-IA+V 1,353Ā±931,353Ā± 93 269.9Ā±18.8269.9Ā± 18.8 251.5Ā±23.4251.5Ā± 23.4 same setting TR-SVO 1,405Ā±3161,405Ā± 316 64.6Ā±37.864.6Ā± 37.8 5.3Ā±12.55.3Ā± 12.5 1,710Ā±1,710Ā± 499 60.6Ā±26.760.6Ā± 26.7 0.0Ā±0.00.0Ā± 0.0 SRA-SVO 1,205Ā±391,205Ā± 39 209.6Ā±9.2209.6Ā± 9.2 83.7Ā±13.083.7Ā± 13.0 same setting SRR-SVO 1,357Ā±1921,357Ā± 192 71.7Ā±32.071.7Ā± 32.0 0.0Ā±0.00.0Ā± 0.0 1,309Ā±2931,309Ā± 293 105.3Ā±40.1105.3Ā± 40.1 15.7Ā±30.815.7Ā± 30.8 Commons Harvest (N=7N=7) Plain PPO 211Ā±2211Ā± 2 30.1Ā±0.230.1Ā± 0.2 27.9Ā±0.727.9Ā± 0.7 same setting TR-EI 657Ā±86657Ā± 86 5.6Ā±1.65.6Ā± 1.6 0.6Ā±0.70.6Ā± 0.7 same setting SRA-EI 416Ā±86416Ā± 86 35.9Ā±3.935.9Ā± 3.9 8.5Ā±4.28.5Ā± 4.2 same setting SRR-EI ā1,604Ā±4,721-1,604Ā± 4,721 15.0Ā±7.115.0Ā± 7.1 ā522Ā±1,235-522Ā± 1,235 214Ā±2214Ā± 2 30.5Ā±0.230.5Ā± 0.2 27.6Ā±1.027.6Ā± 1.0 TR-IA ā1,935Ā±4,576-1,935Ā± 4,576 0.2Ā±0.10.2Ā± 0.1 ā771Ā±1,821-771Ā± 1,821 ā1,678Ā±3,990-1,678Ā± 3,990 1.0Ā±0.61.0Ā± 0.6 ā1,037Ā±2,454-1,037Ā± 2,454 SRA-IA 212Ā±3212Ā± 3 30.2Ā±0.430.2Ā± 0.4 26.7Ā±1.726.7Ā± 1.7 same setting SRR-IA+V 271Ā±66271Ā± 66 15.2Ā±5.315.2Ā± 5.3 0.3Ā±0.70.3Ā± 0.7 209Ā±1209Ā± 1 29.8Ā±0.229.8Ā± 0.2 27.0Ā±1.227.0Ā± 1.2 TR-SVO Ā±744Ā± 128 26.5Ā±12.726.5Ā± 12.7 3.2Ā±7.63.2Ā± 7.6 same setting SRA-SVO 213Ā±6213Ā± 6 30.1Ā±1.330.1Ā± 1.3 24.4Ā±4.624.4Ā± 4.6 same setting SRR-SVO 280Ā±35280Ā± 35 18.7Ā±7.118.7Ā± 7.1 3.5Ā±9.03.5Ā± 9.0 214Ā±4214Ā± 4 30.4Ā±0.630.4Ā± 0.6 26.5Ā±1.726.5Ā± 1.7 Table 6. Commons Harvest, seven agents, eight seeds held out from tuning (4ā11). Values are means Ā± 95% Student-t interval half-widths where shown; bold marks the best per column. In parentheses, the metric that selected the setting (C: collective return; NW: Nash welfare; both: the same setting). Sust.: mean step at which apples are eaten. āMeans of SRA-IA and SRR-IA (both) and of SRR-EI, SRR-SVO, and SRR-IA+V (NW); half-widths ā¤4.2ā¤ 4.2 (C), ā¤0.6ā¤ 0.6 (āNW), and ā¤1.7ā¤ 1.7 (L). L: lowest agent return. Self-scaling control: SVO with Ļ=0Ļ=0, which only rescales the agentās own reward. Method (selection) C āNW L Sust. Plain PPO 211Ā±2211Ā± 2 30.1Ā±0.230.1Ā± 0.2 27.9Ā±0.727.9Ā± 0.7 46Ā±146Ā± 1 TR-EI (both) 657Ā±86657Ā± 86 5.6Ā±1.65.6Ā± 1.6 0.6Ā±0.70.6Ā± 0.7 195Ā±55195Ā± 55 TR-SVO (both) Ā±744Ā± 128 26.5Ā±12.726.5Ā± 12.7 3.2Ā±7.63.2Ā± 7.6 Ā±349Ā± 56 TR-IA (C) ā1,935-1,935 0.20.2 ā771-771 8888 TR-IA (NW) ā1,678-1,678 1.01.0 ā1,037-1,037 2828 SRA-EI (both) 416Ā±86416Ā± 86 35.9Ā±3.935.9Ā± 3.9 8.5Ā±4.28.5Ā± 4.2 195Ā±44195Ā± 44 SRA-SVO (both) 213Ā±6213Ā± 6 30.1Ā±1.330.1Ā± 1.3 24.4Ā±4.624.4Ā± 4.6 52Ā±552Ā± 5 SRR-SVO (C) 280Ā±35280Ā± 35 18.7Ā±7.118.7Ā± 7.1 3.5Ā±9.03.5Ā± 9.0 98Ā±2898Ā± 28 SRR-IA+V (C) 271Ā±66271Ā± 66 15.2Ā±5.315.2Ā± 5.3 0.3Ā±0.70.3Ā± 0.7 77Ā±3177Ā± 31 Five NW selectionsā 209ā214 29.8ā30.5 26.5ā28.2 45ā49 Self-scaling control (both) 302Ā±64302Ā± 64 42.0Ā±6.942.0Ā± 6.9 31.2Ā±2.431.2Ā± 2.4 108Ā±47108Ā± 47 Table 7. Escape Room and Clean Up with five agents, eight held-out seeds. C is collective return, āNW Nash welfare (c=5c=5 in Escape Room, 00 in Clean Up), L the lowest agent return. In parentheses, the metric that selected a setting when the two differ. The last column describes behaviour: the standard deviation of the exit rates in Escape Room, about 0.5 when the same agents always exit and 0 when all exit equally often, and in Clean Up the share of the cleaning done by the two lowest earners, 0.40 for an equal split. Env. Method (selection) C āNW L Behavior Escape Room Plain PPO ā0.25Ā±0.39-0.25Ā± 0.39 4.95Ā±0.084.95Ā± 0.08 ā0.25Ā±0.39-0.25Ā± 0.39 no reliable escape TR-EI (C) 17.00Ā±0.0017.00Ā± 0.00 6.96Ā±0.006.96Ā± 0.00 ā1.00Ā±0.00-1.00Ā± 0.00 concentrated exits, SD 0.49 SRA-EI 15.75Ā±2.9515.75Ā± 2.95 6.78Ā±0.426.78Ā± 0.42 ā1.00Ā±0.00-1.00Ā± 0.00 concentrated exits, SD 0.48 SRR-IA + Standing (NW) 13.08Ā±0.2813.08Ā± 0.28 7.50Ā±0.037.50Ā± 0.03 +1.27Ā±0.12+1.27Ā± 0.12 criterion 8/8, SD 0.13 Clean Up Plain PPO 248Ā±198248Ā± 198 37.0Ā±29.837.0Ā± 29.8 5.4Ā±4.85.4Ā± 4.8 ā TR-SVO (C) 1,405Ā±3161,405Ā± 316 64.6Ā±37.864.6Ā± 37.8 5.3Ā±12.55.3Ā± 12.5 cleaning share 0.86Ā±0.160.86Ā± 0.16 TR-IA 964Ā±107964Ā± 107 178.1Ā±20.6178.1Ā± 20.6 95.9Ā±18.095.9Ā± 18.0 ā SRA-EI 1,286Ā±1411,286Ā± 141 214.4Ā±15.3214.4Ā± 15.3 83.1Ā±16.683.1Ā± 16.6 cleaning share 0.89Ā±0.120.89Ā± 0.12 SRR-EI 1,512Ā±1,512Ā± 140 84.5Ā±29.884.5Ā± 29.8 00 one or two agents only clean SRR-IA+V (NW) 1,353Ā±931,353Ā± 93 269.9Ā±18.8269.9Ā± 18.8 251.5Ā±23.4251.5Ā± 23.4 cleaning share 0.51Ā±0.120.51Ā± 0.12 C.3. Production Is Not Cooperation Collective return says how much the group produces. Figures 8 and 9 show the Nash welfare and the lowest agent return of the same runs, in the same layout, and they change the ranking of the learners. In Clean Up, the true-reward learners lead on collective return under EI and SVO, but in Nash welfare they fall to the bottom of their panels, near 60, while SRA-EI and SRA-SVO climb to about 215 and 210. The lowest agent return separates them further: the true-reward EI and SVO learners leave their poorest agent at zero for the whole run, while SRA lifts it to about 83. In Commons Harvest the reversal is starker. TR-EIās Nash welfare falls to about 6 and its poorest agent to under 1, below plain PPO on both, while SRA-EI stays near 36. The advantage of observing the true rewards lies in production, and it is paid for by the agents who end with nothing. Escape Room shows a ceiling rather than a separation. Every learner that escapes reaches a Nash welfare near 7 and a lowest agent return of ā1-1, because whoever pulls the lever loses a point in every episode. No learner without standing avoids this, which is what Appendix F addresses. Figure 8. Nash welfare over training for the runs of Figure.3 in the main paper. Columns are games and rows are social operators; every learner is at its collective-return selection. Lines are means over eight held-out seeds and shaded regions are pointwise 95% Student-t intervals, with a five-bin moving average. Figure 9. Lowest agent return over training for the runs, with the same layout and intervals as Figure 8. The grey line marks zero. Commons Harvest TR-IA falls below the plotted range, to a final mean of ā771-771, and is marked at the panel edge. C.4. Escape Room at Every Group Size Table 8 asks whether the Escape Room result depends on the group size. At every size from two to twelve agents, at least one self-referenced variant reaches the optimum, so the answer is no: the game stays solvable without the othersā rewards. The last column shows how narrow the path is. Seven of sixteen variants reach the optimum with two agents, but only two or three from five agents on. Self-referenced estimates make the game solvable at every size, but a configuration that works at one size cannot be assumed to work at another. Table 8. Escape Room: collective return as a fraction of the optimum, held-out seeds. āBest self-ref.ā is the best of 16 self-referenced variants (four operators Ć four placements) selected on tuning seeds; the last column counts how many of the 16 reach 90% of the optimum. N M PPO TR Best self-ref. SRA-EI ā„90%ā„ 90\% 2 1 0.00 1.00 1.00 (SRR-EI) 1.00 7/16 3 2 0.00 1.00 1.00 (SRR-IA) 0.69 3/16 5 3 ā-0.01 1.00 1.00 (SRR-IA) 0.93 3/16 7 4 0.00 1.00 1.00 (SRR-IA) 0.82 2/16 10 5 0.00 1.00 1.00 (SRR-IA) 0.69 2/16 12 6 0.00 1.00 1.00 (SRR-IA) 0.00 2/16 Appendix D Where Should the Self-Referenced Estimate Enter? The same self-referenced estimate can enter learning in the reward or in the policy update, and the two places suit different preferences. A preference that compares agents belongs in the reward, with the look-ahead; a benevolent one is safer in the policy update. Two experiments support this division. The first runs the reward placement with and without the look-ahead in every game and under every operator. The second follows both placements as the group grows. D.1. What the Look-Ahead Does Figure 10 runs the reward placement with and without the look-ahead in every game and under every operator. The look-ahead adds the criticās value of the other agentās next observation to its self-referenced trace, Eq.7, so an agent is judged by what it has earned and by what it is about to earn. The look-ahead does two different jobs. In Escape Room under EI and SVO it gets learning started: without it the dashed line stays at zero for the whole run, because agents that never pull the lever never reach the door, their reward model is fitted on transitions with no reward in them, and the social term is zero everywhere. The critic already tells positions apart, and that is enough to push the agents off the selfish solution. Under inequity aversion it prevents a misjudgement: a trace makes an agent that is cleaning look far behind, and an agent averse to being ahead stops eating. Crediting the cleaner with the apples it can expect removes most of that apparent gap, and the collective return in Clean Up doubles. The same mechanism has a cost when the preference only adds outcomes. Under EI in Clean Up, SRR-EI+V produces about 300 apples less than SRR-EI: crediting the others with what they are about to earn pays the agent for a benefit not yet produced. In Commons Harvest the look-ahead does not rescue EI or SVO either; tuning selects the smallest weight we tried, and larger weights stop the agents harvesting. The look-ahead helps a preference that compares agents and can hurt one that adds them up. Figure 10. The value look-ahead. Collective return over training for the reward placement without the look-ahead, SRR, and with it, SRR+V; plain PPO and the true-reward learner are drawn faintly for reference. Every learner is at its collective-return selection; lines are means over eight held-out seeds and shaded regions, drawn for every learner except plain PPO, are pointwise 95% Student-t intervals; where all seeds agree, as for the true-reward learners at the Escape Room optimum, the interval has no width. Learners below the plotted range are marked at the panel edge with their final means. D.2. What Happens as the Group Grows Figure 11 repeats the comparison of placements in Escape Room for two to twelve agents, each cell tuned at its own size, and Figure 12 follows the main learners in all three games and on all three measures. The two methods fail in different ways as the group grows. The policy update dilutes. Its coefficient averages the scores of the Nā1N-1 other agents, Eq.12, so the part that responds to any one agentās action shrinks as 1/(Nā1)1/(N-1). SRA-EI reaches at least two thirds of the optimum up to ten agents and none of it with twelve, and it falls further behind TR-EI as N grows in Escape Room, where it reaches 31 against 45 with ten agents, and in Commons Harvest. The reward placement does not dilute: under inequity aversion SRR and SRR+V sit at the optimum from three agents up to twelve, while SRA-IA stays at zero at every size, because a comparison that only reweights scores does nothing at any N. Inequity aversion also behaves differently with true rewards and with self-referenced estimates. With true rewards it declines as the group grows, TR-IA falling to 3.8 in Escape Room with ten agents and from 820 to 345 apples in Clean Up between three and seven agents, and it collapses in Commons Harvest. With self-referenced estimates and the look-ahead it holds: SRR-IA+V produces most with five agents, gives its poorest agent more than any other learner shown with five and seven, and collapses only with nine agents in Commons Harvest, where every learner with a social term except SRA-SVO collapses. A larger group gives an inequity-averse agent more pairs to be behind in, and the true rewards make each of those comparisons sharper than it is. We have not isolated this mechanism. Figure 11. Where the self-referenced estimate enters, by group size. Escape Room collective return divided by the optimum for each operator, with the estimate in the policy update, SRA, in the reward, SRR, and in the reward with the value look-ahead, SRR+V, against the true-reward learner. Every cell is tuned separately at its N on tuning seeds; points are means over eight held-out seeds and shaded regions, drawn for SRA and SRR+V, are 95% Student-t intervals. Figure 12. Final collective return, Nash welfare, and lowest agent return against the number of agents N for plain PPO, TR-EI, SRA-EI, TR-IA, SRR-IA+V, TR-SVO, and SRA-SVO, each at its collective-return selection for that N; means over eight held-out seeds, bars are 95% t-intervals. The Escape Room optimum grows with N (dotted line). Downward triangles mark means below the plotted range: in Commons Harvest, TR-IA at N=7N=7 ends at ā1,935-1,935, and at N=9N=9 TR-IA, SRR-IA+V, SRA-EI, TR-EI, and TR-SVO end at ā12,818-12,818, ā5,735-5,735, ā3,770-3,770, ā2,268-2,268, and ā1,720-1,720. Commons Harvest Nash welfare and lowest agent return here are the per-step logged values, which differ slightly from the values computed from per-agent means in the other tables. Appendix E Who Pays for Cooperation? A high collective return can hide who receives it and who does the work. In Clean Up the most productive learners split into eaters and cleaners, while SRR-IA+V shares the work. This appendix shows that the pattern holds for all nine learners. It also appears, in its own form, in the other two games, and it survives a change of the selection metric. E.1. All Nine Learners in Clean Up Figure 13 shows all nine Clean Up learners, one block per operator, with every agentās return over training on top and the shares of the apples and the cleaning below. Read down a column, the two rows tell the same story from two sides: an agent whose return line stays near zero is, in the row beneath, the one doing most of the cleaning. The split into eaters and cleaners is the rule, not the exception. In eight of the nine learners the line of the apples and the line of the cleaning cross, and the two poorest agents do between 71% and 100% of the cleaning while the richest does almost none. It forms early and does not change: in the four learners that leave their poorest agent near zero, TR-EI, TR-SVO, SRR-EI+V, and SRR-SVO+V, that agentās line flattens within the first two million frames and never rises. Where the estimate enters matters more than the estimate itself. SRR-EI+V and SRR-SVO+V use the same self-referenced estimates as SRA-EI and SRA-SVO, yet divide the return as unequally as the true-reward learners. The one learner that breaks the pattern is SRR-IA+V: its five return lines climb together, and both of its share lines stay near 20% for every agent, so its high Nash welfare is not an artefact of everyone earning equally little. The Nash welfare puts numbers on this. For every operator our learnerās welfare is far above that of the true-reward learner with the same operator: 214 against 70 under EI, 270 against 178 under inequity aversion, and 210 against 65 under SVO. The gap comes from the poorest agents, not from production, since the collective returns differ by a sixth at most and ours is higher under inequity aversion; a geometric mean is pulled down by the agent with the least, and the true-reward EI and SVO learners leave that agent with almost nothing. The reward placement with EI and SVO is the cautionary case: it uses the same self-referenced estimates as SRA, yet its welfare is no better than the true-reward learnerās, 97 and 81, because its two poorest agents do 84% and 100% of the cleaning. Tables 9 and 10 show that the pattern does not depend on the group size. With seven agents, under SRA-EI the two poorest agents do 86Ā±986Ā± 9% of the cleaning and receive 5% of the apples, while under SRR-IA they do 34Ā±734Ā± 7% and receive 24%. Figure 13. Who receives the return and who does the work for all nine Clean Up learners with five agents. One block per operator; columns are the true-reward learner, SRA, and SRR+V, each at its collective-return selection. Top row of each block: every agentās return over training, agents ranked within each seed by their final return and averaged by rank over the eight held-out seeds, darkest the richest. Bottom row: each agentās final share of the apples, solid, and of the cleaning, dashed, with 95% Student-t intervals; the dotted line is the equal share of 20%. Table 9. Clean Up, three agents, eight held-out seeds (95% CIs). IA settings are Ī±/Ī²Ī±/Ī²; SVO uses Ļ=Ļ/3Ļ=Ļ/3. The last column lists the mean return of the highest through the lowest earner. Method Setting C āNW L Returns Plain PPO ā 373Ā±206373Ā± 206 104.9Ā±61.8104.9Ā± 61.8 48.3Ā±45.248.3Ā± 45.2 172 / 153 / 48 SRA-EI (NW) Ī±=2Ī±=2 768Ā±69768Ā± 69 245.6Ā±24.5245.6Ā± 24.5 172.8Ā±27.9172.8Ā± 27.9 335 / 260 / 173 SRA-SVO Ī±=7Ī±=7 807Ā±36807Ā± 36 258.9Ā±13.5258.9Ā± 13.5 196.1Ā±23.3196.1Ā± 23.3 369 / 242 / 196 SRA-IA 0/10/1 689Ā±32689Ā± 32 228.7Ā±10.7228.7Ā± 10.7 208.6Ā±13.8208.6Ā± 13.8 257 / 223 / 209 SRR-IA+V 0/0.030/0.03 664Ā±213664Ā± 213 211.2Ā±74.1211.2Ā± 74.1 160.3Ā±75.1160.3Ā± 75.1 296 / 207 / 160 TR-IA 0/0.030/0.03 820Ā±37820Ā± 37 265.4Ā±16.1265.4Ā± 16.1 215.7Ā±31.9215.7Ā± 31.9 361 / 243 / 216 TR-EI (C) Ī±=0.01Ī±=0.01 1,010Ā±1,010Ā± 19 62.0Ā±0.962.0Ā± 0.9 00 585 / 425 / 0 Table 10. Clean Up, seven agents, eight held-out seeds (7ā14; 95% CIs). Each learner at the setting selected on the three tuning seeds, by the metric in parentheses. The last column lists the mean return of the highest through the lowest earner. Method Setting C āNW L Returns, high ā low Plain PPO ā 313Ā±193313Ā± 193 33.7Ā±24.033.7Ā± 24.0 5.2Ā±8.25.2Ā± 8.2 70 / 60 / 51 / 49 / 47 / 31 / 5 SRA-EI (NW) Ī±=3Ī±=3 1,953Ā±1741,953Ā± 174 198.5Ā±18.2198.5Ā± 18.2 34.2Ā±13.734.2Ā± 13.7 426 / 409 / 383 / 352 / 285 / 64 / 34 SRA-SVO (NW) Ī±=5Ī±=5 1,356Ā±1111,356Ā± 111 161.8Ā±13.9161.8Ā± 13.9 39.0Ā±9.839.0Ā± 9.8 266 / 252 / 244 / 239 / 231 / 85 / 39 SRR-IA (NW) 0.003/0.10.003/0.1 1,069Ā±1161,069Ā± 116 151.1Ā±16.2151.1Ā± 16.2 116.3Ā±17.8116.3Ā± 17.8 178 / 166 / 162 / 155 / 150 / 142 / 116 SRR-IA+V (NW) 0.003/0.10.003/0.1 1,006Ā±1331,006Ā± 133 140.9Ā±19.8140.9Ā± 19.8 100.3Ā±26.9100.3Ā± 26.9 181 / 166 / 154 / 143 / 133 / 129 / 100 TR-IA (NW) 0/0.030/0.03 345Ā±296345Ā± 296 40.9Ā±37.540.9Ā± 37.5 11.4Ā±14.811.4Ā± 14.8 76 / 65 / 62 / 55 / 52 / 23 / 11 TR-EI (C) Ī±=0.03Ī±=0.03 2,609Ā±2,609Ā± 298 80.1Ā±15.880.1Ā± 15.8 00 568 / 555 / 545 / 541 / 401 / 0 / 0 TR-SVO (C) Ī±=0.01Ī±=0.01 1,369Ā±6231,369Ā± 623 87.6Ā±32.687.6Ā± 32.6 00 276 / 265 / 251 / 245 / 239 / 92 / 0 E.2. The Same Tension in the Other Games Figures 14 and 15 show every agentās return in Escape Room and Commons Harvest. The tension between producing and sharing appears in both, in a form set by the game. In Escape Room it is built in. Every learner that escapes produces the same two-tier outcome, two agents at +10+10 and three at ā1-1, because three of five must hold the lever while the other two walk out. That this is the same under TR-EI, SRA-EI, and SRR-IA+V shows it is a property of the game, not of any learner, and collective return cannot tell a group that rotates the lever from one that does not, since every episode pays the same whoever pulls. This is why the game needs standing, Appendix F. In Commons Harvest it takes the form of a monopoly. Under TR-EI one agent climbs to about 560 of the groupās 657 apples while the rest stay near the floor, so the gameās highest collective return goes almost entirely to one agent. SRA-EI produces less, 416, but its lines stay together and its poorest agent ends at 8 against under 1. The Nash welfare tells the same story more sharply: TR-EIās falls to 6, below even plain PPOās 30, while SRA-EIās is 36. Welfare in this game needs care, though. The self-scaling control, which ignores the others, reaches the highest welfare of all, so in Commons Harvest a high welfare is not by itself evidence of a social preference. Figure 14. Who receives the return in Escape Room with five agents, as in Figure 13. Nash welfare uses the offset c=5c=5 of Eq. (15). Figure 15. Who receives the return in Commons Harvest with seven agents, as in Figure 13. TR-IA collapses and lies below the plotted range. E.3. Selecting on Nash Welfare A natural objection is that our learners only look fairer because of how we selected them. Figure 16 answers it by taking, in each game, the one configuration that each metric selects among all social learners, and plotting every agentās return. The two rows differ in shape, not only in level: the top row, selected on collective return, has a leader and a floor, and the bottom row, selected on Nash welfare, has lines that stay together. In Clean Up the collective-return choice, TR-SVO, produces 1,405 apples with a Nash welfare of 65, and the Nash-welfare choice, SRR-IA+V, produces 1,353 with a welfare of 270, so almost the same harvest is divided very differently. The tuning landscape of Figure 6 shows that this is not one lucky pick: at every group size, the collective-return choice is a true-reward learner and the Nash-welfare choice is one of ours. For a single learner the two metrics usually agree, and where they differ the effect is not consistent. In Escape Room with seven agents, Nash welfare selects SRR-IA with standing at Īŗ=2Īŗ=2, which reaches 72% of the optimum, over Īŗ=1Īŗ=1, which produces more but leaves the poorest agent at a loss. In Clean Up with five agents, the TR-SVO setting chosen on Nash welfare produced more apples on the held-out seeds, 1,710Ā±4991,710Ā± 499, than the one chosen on collective return, 1,405Ā±3161,405Ā± 316, while still leaving one agent with nothing; we report the setting chosen on the tuning seeds and do not reselect. In Commons Harvest the Nash-welfare settings make the SRR learners equal by reducing their production to that of plain PPO, which is the failure mode of the metric. Neither metric is enough alone, which is why we report both and the lowest agent return as well. Figure 16. Agent outcomes under the two tuning objectives. Columns are the three games; rows are the configurations selected on tuning seeds by collective return (top) and Nash welfare (bottom), without the self-scaling controls. Each large panel shows the return of every agent over training: agents are ordered within each seed by their final-window return and then averaged by rank across the eight held-out seeds (darkest: highest final-return rank). The narrow panel shows all eight final seed values and the rank mean Ā± 95% Student-t interval. Insets give the final C and āNW. Ranked curves describe how the return is divided; they do not follow one agent across seeds. Appendix F Standing in the Repeated Game In Escape Room some agents must pay for the lever in every episode, so a fair division is possible only across the episodes of a run. Standing, achieves it: fixed roles become shared ones, at the cost of about a quarter of the optimum. This appendix explains how standing is computed, shows that the result holds on every seed, traces the lost quarter to coordination, and explains why we do not use standing in Clean Up. F.1. How Standing Is Computed Each observer i keeps, for every agent j, a running average GiājG_ij of jās episode return as i estimates it from self-reference, and of its own true return when j=ij=i. After episode k, (23) R^iājk=ātākr^iājt,Ī·k=maxā”(1/k,1/H), R_ij^k= _tā k r_ij^t, _k= (1/k,1/H), (24) Giājk=(1āĪ·k)āGiājkā1+Ī·kāR^iājk,G_ij^k=(1- _k)G_ij^k-1+ _k R_ij^k, a plain mean over the first H episodes and an exponential moving average after that. The mean at the start keeps early episodes from being dominated by the latest one, and the moving average afterwards lets the ledger follow a change in behaviour. We use H=100H=100 episodes in Escape Room and Hā20,30Hā\20,30\ in Clean Up, whose 1,000-step episodes give only a few hundred episodes per environment copy. Every copy of the environment keeps its own ledger, and each entry is updated when that agentās episode ends. From the ledger, i measures how far it has been ahead of the others and how far behind them, ai a_i =1Nā1āājā i[GiāiāGiāj]+,bi=1Nā1āājā i[GiājāGiāi]+, = 1N-1 _jā i[G_i-G_ij]_+, b_i= 1N-1 _jā i[G_ij-G_i]_+, (25) aheadi _i =aiai+ci,behindi=bibi+ci, = a_ia_i+c_i, _i= b_ib_i+c_i, with the scale ci=|Giāi|+(Nā1)ā1āājā i|Giāj|c_i=|G_i|+(N-1)^-1 _jā i|G_ij| plus a small constant. Dividing by a scale built from the returns themselves lets one gate strength Īŗ work in games whose returns differ by two orders of magnitude: both fractions lie in [0,1)[0,1) and do not change when every return is multiplied by the same constant. Both are zero when iās history is level with the othersā, so the gate leaves a fair division alone and only separates āalways meā from āeach of us in turnā. The gate is off at the start of training, and this matters. It waits for the reward-model warm-up W and a further gate warm-up, and then acts only once some agentās average exceeds a threshold, two per episode in every Escape Room run. Before that the group has nothing to share: a gate applied from the start discounts the first agent that earns anything, and in our runs it prevents cooperation from forming at all. The threshold is tested on all agents, the observer included, because with three agents and two pulling the lever only one agent collects, and testing the others alone would never switch the gate on for the agent that is ahead. F.2. The Result Holds on Every Seed Figure 17 shows the four pairs of configurations that differ only in standing: SRA-EI with three and five agents and SRR-IA with five and seven. Panels (a) and (b) are the result. Without standing the exit rates split within the first 150 thousand frames and stay split: two agents exit in every episode and three in none, on every seed. With standing no line sits at zero, every agent exits in at least a quarter of the episodes, and the lines gather around the even share of two in five. Panel (c) gives the price, a collective return that settles at about three quarters of the optimum, and panels (d)ā(g) show the same trade in all four pairs. A mean can hide a mixture of outcomes, so Figure 18 shows every held-out seed. Without standing every seed settles at a loss of one point for its worst-off agent; with standing the worst-off agent is above zero on almost every seed. Table 11 counts them: standing takes every pair from no shared seed to six or eight of eight. A larger sample confirms the two SRA-EI pairs. Over twenty seeds the group with three agents reaches 5.62Ā±0.375.62Ā± 0.37 of an optimum of 8, its lowest agent gains 1.48Ā±0.301.48Ā± 0.30 per episode, and the agents exit on 27, 28, and 30% of episodes, close to the 33% of an even split. With five agents the group reaches 13.56Ā±0.1013.56Ā± 0.10 of 17 and exit rates lie between 35 and 41%, against 40% for an even split. These are frequencies over many episodes, not an order in which the agents take turns. Figure 17. Standing in the repeated game, Escape Room. (a, b) SRR-IA with five agents, without and with standing, the only difference between the two: the fraction of episodes in which each agent exits, ranked within each seed by its final exit rate and averaged by rank over the eight held-out seeds; the dotted line is the even share, two exits in five. (c) Collective return divided by the optimum for the same two runs. (dāg) Lowest agent return over training without and with standing for the four exact pairs. Means over seeds 11ā18 with pointwise 95% Student-t intervals. Figure 18 shows every seed. Figure 18. Every held-out seed of the standing pairs in Escape Room over training: the lowest agent return, top, and the collective return over the optimum, bottom, without standing in grey and with it in colour. Thin lines are single seeds, seeds 11ā18, and bold lines their mean. Without standing, every seed settles at a loss of one point for its worst-off agent, and most groups reach the optimum; with standing, the worst-off agent stays above zero on almost every seed, and the group settles at 70 to 80% of the optimum. ā The two TR-IA pairs also differ in their PPO settings. Table 11. The four exact pairs of Figure 17 and the two descriptive TR-IA pairs, held-out seeds 11ā18. Values for the with-standing arm are means Ā± 95% Student-t interval half-widths. A seed counts as shared when Ls>0L_s>0 and the exit rates have a standard deviation below 0.20; the last column gives the shared seeds without ā with standing. Every without-standing arm is 0/8. ā The TR-IA pairs also differ in their PPO settings. ER(N,M), with-standing configuration C/CāC/C LsL_s Exit SD Shared seeds (3,2) SRA-EI, Īŗ=4Īŗ=4 0.711Ā±0.0350.711Ā± 0.035 1.514Ā±0.2201.514Ā± 0.220 0.030Ā±0.0150.030Ā± 0.015 0/8ā8/80/8ā 8/8 (5,3) SRA-EI, Īŗ=4Īŗ=4 0.797Ā±0.0020.797Ā± 0.002 0.114Ā±0.1190.114Ā± 0.119 0.129Ā±0.0030.129Ā± 0.003 0/8ā6/80/8ā 6/8 (5,3) SRR-IA, Īŗ=1Īŗ=1 0.769Ā±0.0160.769Ā± 0.016 1.268Ā±0.1221.268Ā± 0.122 0.131Ā±0.0160.131Ā± 0.016 0/8ā8/80/8ā 8/8 (5,3) TR-IA, Īŗ=1Īŗ=1ā 0.735Ā±0.0320.735Ā± 0.032 0.942Ā±0.4030.942Ā± 0.403 0.107Ā±0.0200.107Ā± 0.020 0/8ā8/80/8ā 8/8 (7,4) SRR-IA, Īŗ=2Īŗ=2 0.718Ā±0.0390.718Ā± 0.039 1.201Ā±0.2821.201Ā± 0.282 0.104Ā±0.0400.104Ā± 0.040 0/8ā8/80/8ā 8/8 (7,4) TR-IA, Īŗ=2Īŗ=2ā 0.691Ā±0.1340.691Ā± 0.134 0.476Ā±0.7830.476Ā± 0.783 0.122Ā±0.0500.122Ā± 0.050 0/8ā6/80/8ā 6/8 F.3. Why Sharing Costs a Quarter of the Optimum With standing, every configuration reaches between 69% and 80% of the optimum. Escape Room is solved in one step only if exactly the right number of agents pull the lever at once; too few and the door stays shut, too many and the group pays for nothing. Once every agent sometimes pulls, the question is whether they manage to pull in the right numbers. Figure 19 shows that they do not coordinate. The number of agents that pull at the first step is what independent choices would give. With five agents and three needed, each agent pulls first with probability pĀÆ=0.35 p=0.35, independent choices give exactly three pullers with probability (53)āpĀÆ3ā(1āpĀÆ)2=0.18 53 p^3(1- p)^2=0.18, and the observed share is 0.18. With three agents and two needed, pĀÆ=0.69 p=0.69 gives 0.44 against the observed 0.46. The lost quarter is the cost of that mismatch. Figure 19. How many agents pull the lever at the first step of an episode in Escape Room with standing, over 400 evaluation episodes per run, for the four configurations with standing. Bars are means over training seeds and dots single seeds. The line is the binomial distribution that independent choices with the observed pull rate would give; the episode is solved at once only when exactly M agents pull. If the agents knew more, would they coordinate better? Table 12 tests three ways of giving them more information, for the selected SRA-EI learner. Training the critic used in the cross scores on raw returns lowers the return. Giving the policy its own standing as an input, so that it can act on how far ahead it has been, changes little. Replacing the private self-referenced ledgers with one ledger of true returns shared by all agents, which removes both the estimation error and the disagreement between ledgers, also changes little. None of the three recovers the lost quarter, so what is missing is a way to agree on who pulls this time, not a better estimate of who has been paying. The standing input is the only control that goes beyond decentralized execution, because the policy then sees a quantity built from the other agentsā transitions while it acts. It enters the policy and the critic through a layer initialized to zero, (26) hāhā(1+WĪ³āsi)+WĪ²āsi,si=(aheadi,behindi),hā h (1+W_Ī³s_i)+W_Ī²s_i, s_i=(ahead_i,behind_i), so the network is exactly the unconditioned one until the agent has a standing, and the standing is updated after every step so that the next action can depend on it. The shared true ledger keeps the reward models and the self-referenced social term and replaces only the ledger, so it changes two things at once, and its null result is weaker evidence than it looks. Table 12. Collective return of the selected SRA-EI learner in Escape Room, without and with each control, and without or with the standing gate at Īŗ=4Īŗ=4; three seeds per cell. The optimum is 8, 17, and 26 at N=3N=3, 5, and 7. Control Gate N=3N=3 N=5N=5 N=7N=7 Critic trained on raw returns off 8.00ā7.448.00ā 7.44 15.91ā13.6215.91ā 13.62 21.60ā19.9721.60ā 19.97 on 5.98ā4.675.98ā 4.67 13.52ā13.0313.52ā 13.03 ā Standing as a policy input off 8.00ā8.008.00ā 8.00 15.91ā15.9915.91ā 15.99 21.60ā21.4321.60ā 21.43 on 5.98ā6.405.98ā 6.40 13.52ā13.5313.52ā 13.53 ā One shared true ledger on 5.98ā5.705.98ā 5.70 13.52ā13.4913.52ā 13.49 ā with standing input on 6.40ā6.176.40ā 6.17 13.53ā13.5413.53ā 13.54 ā F.4. Why Not Clean Up Standing is a mechanism for a role that can be taken in turns, and Clean Up does not have one: cleaning is continuous work. The learner that divides Clean Up best already shares that work, since SRR-IA+Vās two poorest agents do 51% of the cleaning, Figure 13. We therefore tried standing on the learner that concentrates the work, SRA-EI with five agents, varying its strength, horizon, aggregate, and source. On the three tuning seeds no setting improved on SRA-EI without standing, which produces 1,551 apples with a Nash welfare of 227 and a lowest agent return of 78. The best of the 24 settings reaches a Nash welfare of 249 and a lowest agent return of 86, within the spread between seeds, and the four strongest settings lower the collective return to between 481 and 830 without helping the poorest agent. No setting passed selection, so none was run on held-out seeds. Appendix G What if Agents See Only Part of the Map? With a 15Ć1515Ć 15 window, agent i must assess jās situation from jās own small window, a part of the map that i may never have seen from there. The two placements then part ways. The policy update keeps its cooperation, while inequity aversion in the reward collapses. The collapse comes from the look-ahead, not from the self-referenced estimates. This appendix gives the evidence for each step and asks whether longer training changes the picture in Commons Harvest. G.1. The Policy Update Keeps Cooperating With feed-forward agents in Clean Up, SRA-EI produces about 1,280 apples, as with the full view, against about 1,900 for TR-EI, which leaves its poorest agent with nothing. With recurrent agents SRA-EI rises to 1,725 apples and the highest Nash welfare of all learners, 224, while TR-EI produces 2,196 and again leaves its poorest agent at zero: the same split into eaters and cleaners as with the full view, now with memory. In Commons Harvest SRA-EI produces 646 apples with feed-forward agents and 482 with recurrent ones, against 187 for plain PPO. These Harvest runs train for 7 million frames and are retuned, so they are not directly comparable with the full view. Plain PPO with a GRU fails in Clean Up at its selected setting, earning three apples, which shows the risk of selecting on two tuning seeds. G.2. Why Inequity Aversion Collapses SRR-IA+V earns nothing in Clean Up with feed-forward agents, recovers only partly with recurrent ones, to about 400 apples, and stays at plain PPOās level in Commons Harvest. The self-referenced estimates do not explain this. Their correlation with the true rewards stays between 0.98 and 0.99 with the small window, as high as with the full view. Even the agents that earn almost nothing estimate their few rewards with a mean error of 0.01. Two controls with feed-forward agents in Clean Up isolate the look-ahead. SRR-IA with the same weights but without the look-ahead produces 111 apples on average instead of none. TR-IA+V, which uses the true rewards together with the look-ahead, produces nothing at all, while TR-IA without it reaches 573 apples at its own selected setting. With true rewards and self-referenced estimates alike, adding the look-ahead under a partial view destroys cooperation. The look-ahead adds the criticās value of the other agentās next observation, and under a partial view that observation shows a different part of the map from anything the critic has learned to value from there, so the levels that inequity aversion compares are misjudged and the agent reacts to a gap that is not there. G.3. Longer Training in Commons Harvest Recurrent agents in Commons Harvest produce less than feed-forward ones, so we asked whether they were simply undertrained. Figure 20 reruns the candidate settings for 14 million frames, twice the length of the other partial-view runs. Our learners barely change after 7 million frames: SRA-EI, SRA-SVO, and SRR-IA+V stay close to the level they had reached, and SRR-IA+V with the lower entropy collapses into beam fights. Only the true-reward EI learner keeps improving, in both collective return and Nash welfare, so the extra training widens the gap rather than closing it. The difficulty with recurrent agents in this game is therefore not a matter of training time. It is also the one setting in which our fairness advantage disappears: there, the fairer setting of TR-EI leaves its poorest agent better off than any of ours. Figure 20. Commons Harvest with recurrent agents and the partial view, trained for 14 million frames; the dotted line marks 7 million, the length of the runs in Figure.4 in the main paper. Rows are operators and columns the collective return, Nash welfare, and lowest agent return; each line is one candidate setting, solid and dashed for the two settings of a learner, means over the eight held-out seeds with pointwise 95% Student-t intervals. One TR-SVO setting had not finished when the figure was made. Appendix H Agents That Value Apples Differently We now deliberately violate the condition under which self-reference can recover another agentās reward. Two of five Clean Up agents value an apple at 2 rather than 1. A self-referenced agent therefore cannot recover the otherās true reward from its transition: it assigns the transition the value that it would itself receive. The experiment asks whether self-reference remains useful when it no longer estimates the otherās private reward, but only assesses the otherās situation from the agentās own perspective. Every learner keeps the setting selected under shared rewards. Figure 21 shows every agentās return, with the true-reward learner in the left column, ours in the right, and one row per operator. In every row the right panelās lines sit closer together than the left panelās: our learners divide the return more evenly, reaching a higher Nash welfare and leaving the poorest agent more, 101 against 1 under EI, 224 against 117 under inequity aversion, and 121 against 16 under SVO. The ranked lines cannot show which agents value apples more, so Figure 22 splits each learnerās agents into the two high-value agents and the three others. The split shows that no learner divides by value. In every learner, including the three given the true rewards, the high-value agents eat about as many apples as the others, and so end with about twice the return. The true-reward learners lean slightly towards leaving the cleaning to the low-value agents, whose apples are worth less, but the spread between seeds is too wide to call it an effect. What separates the learners is consistency. SRR-IA+V gives every agent nearly the same apples and the same share of the work on every seed, while the true-reward learners swing widely, and on some seeds a single agent eats more than 700 apples. The doubled return of the high-value agents is therefore not a consequence of self-reference, since agents that observe the values produce it too. Self-referenced agents cannot see the difference: each assesses every apple at its own value. Here, however, the agents that can see the difference do not act on it either, so the counterfactual assessment costs nothing that the true rewards would have gained. Self-reference stays useful when it is no longer estimating the otherās private reward. A real test of whether judging others by oneās own values misleads an agent would need a preference that responds to the value gap when it is visible, such as inequity aversion on true rewards with a strong penalty on being ahead, which the selection for shared rewards did not choose. Figure 21. Who receives the return in Clean Up when agents 2 and 4 earn two points per apple. Rows are operators and columns the true-reward learner and ours, each at the setting selected under shared rewards. Each line is one agentās return over training, ranked within each seed from the highest to the lowest earner and averaged by rank over the eight held-out seeds. Our learners reach a higher Nash welfare and leave the poorest agent more than the true-reward learner of the same operator; under SRR-IA+V the agents eat almost equally many apples, so the two agents with the higher value end with about twice the return of the others. Figure 22. What each learner gives the agents that value apples more, in Clean Up with five agents when agents 2 and 4 value an apple at 2. One column per operator, comparing the true-reward learner with ours, each at the setting selected under shared rewards. Top: return per agent over training for the two high-value agents, solid, and the three others, dashed, means over the eight held-out seeds with pointwise 95% Student-t intervals. Middle and bottom: final apples eaten and share of the cleaning per agent, dark for the high-value agents and light for the others, with 20% an equal share of the cleaning; bars are means with 95% intervals and dots single seeds. Appendix I Limitations Our conclusions hold within the limits of the study, which we collect here. The shared reward function. A self-referenced estimate recovers another agentās reward only when every agent is rewarded for the same events in the same way. When values differ, it becomes a counterfactual assessment: the reward that the agent itself would receive in the otherās situation. We test one such case, agents that value an apple differently (Appendix H). There no learner acts on the difference, so the test cannot show whether self-reference would mislead an agent that otherwise would. Agents whose rewards come from different events, or from events that their transitions do not show, are beyond a model of oneās own reward. One learning algorithm. Every learner is independent PPO. The reward placement needs only a reward and the policy-update placement only a weighted policy gradient, so both should carry over to other actor-critic methods, and the reward placement to value-based learners, but we have not tested either. Selection on few seeds. Every setting is chosen on two or three tuning seeds by the mean minus one standard deviation, a rule of thumb rather than a confidence bound. Its failures are visible: plain PPO with a recurrent policy, selected this way, earns three apples in Clean Up with the partial view (Appendix G.1), and one TR-SVO setting produced more on the held-out seeds than the setting selected for it (Appendix E.3). Because every number comes from held-out seeds, these failures add noise to the comparison but do not favour any learner. Selected learners, not isolated placements. The two placements are compared at their selected settings, which differ in their social weights and PPO settings as well as in where the estimate enters. The comparison therefore describes the learners we would deploy rather than isolating the placement, and the mechanisms we give for it, such as why the true rewards hinder inequity aversion in Clean Up and as the group grows, are explanations we have not tested directly. Estimation in Commons Harvest. In Commons Harvest the reward model is a small head on the policyās features rather than a model of the transition (Appendix B.2), and the large beam penalty dominates what it learns. This is the game where our learners fall furthest short of the true rewards, and a better model of the transition may narrow the gap. Welfare as a measure. The Nash welfare rewards balance, and it can be raised without cooperation: in Commons Harvest a control that ignores the others reaches the highest welfare of all (Appendix C.2). We therefore read it together with the collective and the lowest agent return, and never alone. Cost. Every agent evaluates its reward model and critic on every other agentās stream, which costs ā”(N2)O(N^2) forward passes per step. This is affordable for the group sizes we study, up to twelve agents, but it grows quickly beyond them. Appendix J Extended Related Work Our work sits where three lines of research meet: social preferences that help learning agents cooperate, models of other agents built from oneās own, and the information that multi-agent methods assume they can read from the other agents. We review each, and close with fairness, which we use to evaluate rather than to train. Table 13 summarizes what each family of methods reads from the others. Sequential social dilemmas and social preferences. Sequential social dilemmas extend matrix-game dilemmas to environments in which cooperation and defection are policies rather than single actions (Leibo et al., 2017); common-pool resource games such as Commons Harvest show how independent learners deplete a shared resource (Perolat et al., 2017), and Melting Pot collects many such games into a benchmark suite (Leibo et al., 2021; Agapiou et al., 2023). Social preferences are the most direct remedy. Hughes et al. (Hughes et al., 2018) give agents the inequity aversion of Fehr and Schmidt (Fehr and Schmidt, 1999), computed on traces of every agentās reward, and show that aversion to being ahead sustains cleaning in Clean Up while aversion to being behind sustains punishment in Commons Harvest; their agent is our TR-IA. McKee et al. (McKee et al., 2020) give agents social value orientations and study how the diversity of orientations in a population shapes its behaviour. Peysakhovich and Lerer (Peysakhovich and Lerer, 2018) show that prosocial agents, which add the othersā rewards to their own, find the cooperative equilibrium of stag hunts more often than selfish ones. Wei et al. (Wei et al., 2026) combine altruistic and fairness preferences through a constant-elasticity-of- substitution (CES) utility and apply it to expected future returns, providing a forward-looking social utility. Their utility still takes other agentsā returns as inputs; we instead replace the unavailable peer-outcome input with a self-referenced estimate. Further work has learned reciprocity (Eccles et al., 2019), evolved intrinsic social motivations (Wang et al., 2019), and compared moral preferences in the same dilemmas (Tennant et al., 2023). Conditional cooperation, as in approximate tit-for-tat (Lerer and Peysakhovich, 2018), trains a cooperative policy on the sum of all agentsā rewards and switches to it when the partner cooperates. These approaches generally rely on other agentsā rewards, outcomes, or welfare signals during training. We keep their preferences unchanged and replace only that input. Changing the incentives. A second line changes the rewards agents receive rather than what they value. Agents can gift part of their reward (Lupu and Precup, 2020), learn to pay incentives that change how others learn (Yang et al., 2020), share rewards along a network (Yi et al., 2022), learn how much of the othersā losses to mix into their own to reduce the price of anarchy (Gemp et al., 2022), agree on enforceable contracts (Haupt et al., 2024), or transfer the smallest amount of reward that makes cooperation stable (Willis et al., 2024). These mechanisms alter or transfer the rewards available to agents, rather than infer another agentās outcome from its observed experience. Vinitsky et al. (Vinitsky et al., 2023) take a different route that is close in spirit to ours: their agents learn which behaviours a group sanctions by watching public sanctions, a social signal learned from observation rather than read from rewards. Social influence (Jaques et al., 2019) also achieves coordination without access to othersā rewards by rewarding causal influence over their actions; its decentralized variant can compute this signal from observed actions using a learned model of other agents. Unlike our method, however, influence is the social signal itself rather than a representation of the otherās outcome. Conventions can also emerge among agents with different preferences without social influence (Kƶster et al., 2020). What cooperative methods read from the others. Cooperative multi-agent reinforcement learning usually assumes a team reward. Value decomposition learns one value per agent that adds up (Sunehag et al., 2018) or combines monotonically (Rashid et al., 2018) into the teamās value, and centralized training with decentralized execution gives each agent a critic that sees the othersā observations and actions during training (Lowe et al., 2017; Foerster et al., 2018b; Yu et al., 2022). Counterfactual baselines (Foerster et al., 2018b) and difference rewards (Wolpert and Tumer, 2001) assign the team reward to individual agents, and opponent shaping (Foerster et al., 2018a) differentiates through the other agentās learning step. Our setting shares what centralized training shares, the observations and actions, but in a general-sum game with no team reward and no access to the othersā rewards, critics, or gradients. SRA resembles a counterfactual baseline in form, since it weights an agentās update by how the othersā transitions went, but it uses no joint action, no shared reward, and no gradient into another agent. Independent PPO, which we build on, is a strong baseline in this family (de Witt et al., 2020; Yu et al., 2022). SRR is a form of reward shaping, but not a potential-based one (Ng et al., 1999): it is meant to change which policies are optimal, as every social preference does. Modelling others from oneās own experience. Agents that model other agents are surveyed by Albrecht and Stone (Albrecht and Stone, 2018). Deep opponent models learn to predict the othersā actions from their observations (He et al., 2016), and machine theory of mind learns to predict another agentās behaviour and false beliefs from a few observations (Rabinowitz et al., 2018). Closest to our view of empathy are methods that use the agentās own machinery to understand another. Self Other-Modeling infers another agentās goal by running the agentās own policy on the otherās observations (Raileanu et al., 2018). Empathic deep Q-learning evaluates the agentās own Q-function from the otherās perspective to avoid harming it (Bussmann et al., 2019). Sympathy-based agents infer another agentās reward and incorporate it into their own objective (Senadeera et al., 2022); EMOTE instead uses the learnerās own action-value function as a reference for modelling the other agent (Senadeera et al., 2024). LASE uses perspective taking to decide how much reward to give away to a co-player (Kong et al., 2024), and homeostaticcoupling models prosocial behavior through shared or inferred homeostatic states (Yoshida and Man, 2025).. We share their premise that oneās own experience is the natural model of anotherās, and differ in what we do with it: we assess the otherās transition with a reward model fitted on the agentās own transitions, feed that self-referenced estimate to standard social preferences, and ask where it should enter learning. Inferring rewards from behaviour. Inverse reinforcement learning infers rewards from behavior under assumptions about the otherās decision process (Ng and Russell, 2000), and has been extended to multi-agent settings (Natarajan et al., 2010; Yu et al., 2019); cooperative inverse reinforcement learning lets a robot learn a humanās reward while acting with them (Hadfield-Menell et al., 2016). In cognitive science, inverse planning infers othersā goals and desires from their behavior under an assumed planning model (Baker et al., 2009; Jara-Ettinger, 2019), and related models infer higher-level intentions in social interaction (Kleiman-Weiner et al., 2016). These approaches infer latent preferences, goals, or intentions from what the other does. Self-reference instead evaluates what happens to the other using the agentās own reward model. Private rewards and communication. Some work keeps rewards private but lets agents exchange other signals. In suggestion sharing, agents exchange suggested actions instead of rewards or values (Jin et al., 2025), and in multi-agent reward prediction a reward model trained on external, episode-level assessments of the groupās outcome steers decentralized agents towards goals such as sustainability and equality (Caftory et al., 2026). Our agents exchange nothing and receive no evaluation beyond their own reward. Empathy and social preferences in people. Social preferences have long been studied in people, particularly in laboratory games (Fehr and Schmidt, 1999; Bolton and Ockenfels, 2000; Charness and Rabin, 2002; Fehr and GƤchter, 2000; Camerer, 2003) and across societies (Henrich et al., 2001). Psychology separates the cognitive part of empathy, estimating anotherās state, from its motivational part, caring about it (Batson, 1991; Decety and Jackson, 2004; de Waal, 2008). The estimate is explained by mapping the otherās situation onto oneās own representations (Preston and de Waal, 2002), by simulating oneself in the otherās place (Gordon, 1986; Goldman, 2006), or by inverse planning (Baker et al., 2009; Jara-Ettinger, 2019); self-reference follows the simulation route. Simulation has a known cost: people assume that others share their preferences (Ross et al., 1977) and adjust too little from their own perspective (Epley et al., 2004). Our experiment with different values in Appendix H is a first test of whether learning agents inherit that bias. Cooperation across repeated encounters, which our standing ledger addresses, has a long history of its own, beginning with the evolution of reciprocal strategies (Axelrod, 1984). Fairness. Fairness has been built into multi-agent learning by centralized mechanisms (Grupen et al., 2022), by optimizing welfare functions of the agentsā returns (Jiang and Lu, 2019; Zimmer et al., 2021; Siddique et al., 2020), by comparing egoistic, utilitarian, and egalitarian objectives (Dong et al., 2024), and, in asymmetric sequential social dilemmas, by correcting social incentives so that agents that differ are treated fairly rather than equally (Demir et al., 2026). All of these need the agentsā returns. We evaluate fairness using: the Nash welfare (Kaneko and Nakamura, 1979), which rewards both production and balance and has strong fairness properties in allocation (Caragiannis et al., 2019), and the lowest agent return, the maximin criterion of Rawls (Rawls, 1971). Table 13. What each method uses from the other agents during training (ā : used; ā : in some variants): rewards, observations and actions, policies or gradients, and a centralized critic or joint value. At execution every method listed acts on the agentās own observation. Method Rewards Obs./act. Policies Critic Inequity aversion (Hughes et al., 2018) ā Social value orientation (McKee et al., 2020) ā Prosocial reward (Peysakhovich and Lerer, 2018) ā VDN, QMIX (Sunehag et al., 2018; Rashid et al., 2018) ā (team) ā ā COMA (Foerster et al., 2018b) ā (team) ā ā MADDPG (Lowe et al., 2017) ā ā ā LOLA (Foerster et al., 2018a) ā ā ā Social influence (Jaques et al., 2019) ā ā SRR, SRA (ours) ā The table makes the position of our learners plain. The social-preference methods need only the othersā rewards, which is exactly the channel we remove. The cooperative and centralized methods read observations and actions as we do, but also a team reward, a centralized critic, or the othersā policies and gradients. Our learners occupy the one row that reads observations and actions alone, so they ask for less than centralized training with decentralized execution and, unlike the social preferences, for nothing that an agent could not have obtained by watching.